abstract: A developer of mobile or desktop applications is responsible for implementing the network logic of his software. Nonetheless: i) Developers are not network specialists, while pressure for emphasis on the visible application parts places the network logic out of the coding focus. Moreover, computer networks undergo evolution at paces that developers may not follow. ii) From the network resource provider point of view, marketing novel services and involving a broad audience is also challenge for the same reason. Moreover, the objectives of end-user networking logic are neither clear nor uniform. This constitutes the central optimization of network resources an additional challenge. As a solution to these problems, we propose the Socket Store. The Store is a marketplace containing end-user network logic in modular form. The Store modules act as intelligent mediators between the end-user and the network resources. Each module has a clear, specialized objective, such as connecting two clients over the Internet while avoiding transit networks suspicious for eavesdropping. The Store is populated and peer-reviewed by network specialists, whose motive is the visibility, practical applicability and monetization potential of their work. A developer first purchases access to a given socket module. Subsequently, he incorporates it to his applications under development, obtaining state-of-the-art performance with trivial coding burden. A full Store prototype is implemented and a critical data streaming module is evaluated as a driving case. [^1]
author: Christos Liaskos, Ageliki Tsioliaridou, Sotiris Ioannidis
Foundation for Research and Technology - Hellas (FORTH)
Emails: {cliaskos,atsiolia,email@example.com
title: The Socket Store: An App Model for the Application-Network Interaction

# Introduction

and academic research in computer networks has bloomed in the past decades. Sub-disciplines have been defined, and state-of-the-art solutions have been proposed for almost every conceivable topic. A huge knowledge potential has been accumulated, which is kept up to date with scientific breakthroughs, culminating with the recent advancements in virtualization and Software-Defined Networking (SDN). The interested networks expert can use mature classification systems provided by formal publications bodies, become current with latest advancements and innovate in platforms that are more than ever modular and hardware-decoupled.

Switch you point of view to that of the average developer of mobile or desktop applications. You have a promising idea for a new *app* that will provide a novel utility in every-day life. You need a graphical interface that stands out and multi-platform support, whilst being pressed for minimal development time. What part of your coding effort will be devoted to "under-the-hood" aspects, i.e., to proper network programming? Moreover, how accessible is the network research knowledge pool to you? Assuming that the proper time is invested, will you also consider the newest network infrastructure capabilities or reported issues?

Expectedly, the problem of unused research knowledge potential exists and has important consequences. Specifically, surveys identify the inefficient app-side communication programming as the top factor impacting both application and network performance . Moreover, in a 2016 study, a $66\%$ percentage of queried professionals acknowledged difficulty in keeping up with the advances in the networking field . These studies exhibit: i) a clear lack of networking code sophistication during app development, and ii) unwillingness to overcome this problem via educative means or additional development effort. A partial solution to the problem are client-side libraries that have appeared, which expose simple programming interfaces and hide the networking complexity . Despite their high importance, such solutions seek to provide basic connectivity, and not research state-of-the-art performance or advanced functionality. Moreover, given that they constitute app-side solutions, the goal of close interaction between applications and the network remains elusive at large, despite the rapid advances in related infrastructure capabilities.

From the network resource provider aspect, inefficient client-side networking code can also be problematic. Without close interaction with the app, the network must strive to equally serve both sane and insane app-side logic. The latter may comprise buggy, greedy or malevolent behavior, allowing for, e.g., DoS attacks . Moreover, sophisticated services offered by network and resource providers, such as Infrastructure-as-a-Service, virtualized functions and protocol stacks, require a good degree of specialization , limiting their use to well-trained network specialists. Companies are already seeking the involvement of people from the app development world, in order to increase the commercial exposure of such services and encourage their further evolution .

The present work contributes the Socket Store, a novel way for connecting the research knowledge potential to the software development world. The Store is platform and marketplace involving network specialists, developers and network resource providers, each with well-defined, fitting roles and motivation. Within the Store, network resource providers expose the infrastructure capabilities, and specialists publish reusable, network logic modules that operate on top of them. Developers browse the Store, purchase access to modules fitting their application, and simply invoke them with minimal coding . Internally, the Store uses a universal and natural modeling approach: Multi-agents are employed to model all network resources, network logic modules, the applications and all their cross-interactions. Through the Store specialists gain visibility, automatic and fair performance evaluation of their contributions, as well as monetization potential of their research. Network resource providers gain a clear, uniform understanding of the preferences of their end-clients, network optimization potential and better commercial visibility of advanced services. Finally, the developers can minimize their expenses by cutting down development time, while obtaining network performance at the level of research state-of-the-art.

The paper is organized as follows. Section  provides a high-level view of the Store workflow. Section  gives an introduction to multi-agent systems. The Store architecture is detailed in Section . A full prototype of the Store is implemented and evaluated in an SDN environment in Section . Section  further discusses what the Store can offer and what it does not. The conclusion is given in Section .

# System Overview

The Socket Store is a market place where software developers without specialty on computer networks can purchase intelligent network code. This code is intended to offer high performance and serve specialized tasks, while necessarily being easy to use and well-tested. Thus, the Store provides:

-   A uniform and natural model to represent the interaction between an application and the network. An agent-based modeling is employed, which is generic and hides the network-related specifics from the developer.

-   A robust code review process, which is strongly tied to the academic research evaluation process.

-   Access to testbeds for automatic and fair network code evaluation and ranking.

The Store refers to the interaction of software applications with existing network resources, prioritizing compatibility. However, it allows network resource provides to publish novel, specialized services as well, promoting the evolution of networking.

## Involved Parties

The Store involves three categories of professionals. The Software Developers, the Network Specialists and the Network Resource Providers.

The first group (Developers) refers to any individual or company that develops software in the broad sense. Emphasis is placed on professionals that develop end-user mobile or desktop applications. In many such cases, where an end-user interacts directly with the software, it can be more significant for the developer to focus his efforts on the end-user's visual and auditory experience. Under-the-hood concerns, such as optimizing the interaction between the application and the network resources, can constitute a considerable overhead in terms of development time and investment in obtaining the required know-how.

The Network Specialists group represents the individuals or companies that specialize in developing networking solutions. The group can comprise academic and industrial research teams, which see interest in pinpointing network performance bottlenecks or opportunities, performing thorough studies of related work, and providing state-of-the-art contributions. All areas of specialty that can be practically applied are considered, exemplary including quality of service, security, energy efficiency and resource sharing.

The Network Resource Providers is a broad term comprising privately help companies or the public, offering physical, virtual or logical resources. Companies may specialize in Cloud provision, secure overlay networking, Internet state monitoring services via passive/active measurements and more. Providers can choose to actively participate to the Socket Store, facilitating the use of resources by the Specialists . However, this approach is not mandatory since the Specialists may themselves model and use existing resources and services.

Finally, the Socket Store is transparent to plain users, who download and use applications normally.

## Workflow

The workflow involving the Store and each party is shown in Fig. . The Developers are considered to be in the process of creating or maintaining a network-enabled application. Instead of programming the interaction with the network themselves, Developers browse the Store for network modules that fit their needs. Subsequently, they purchase and incorporate it to the developed application, and proceed to push it to their distribution channels. The term "Purchase" has the same meaning as in app stores: obtaining for a monetary price or for free. Enabling modules as in-app purchases is also allowed.

The Specialists are responsible for creating innovative and competitive Socket modules. To this end, they combine existing resource agents or create new agent-based representations of resources. A module undergoes a submission, followed by a cycle of reviewing by other researchers, and revisions performed by the original module creators. Moreover, each submitted module is required to be associated with one or more Store performance metrics, in order to be automatically evaluated in testbeds and ensure its conformity with the promised functionality. Upon successful revision, a module is accepted for publication at the Store, becoming browse-able and usable by Developers.

The workflow of Resource providers is not necessarily altered. The providers continue to normally offer access to their resources, without being obliged to offer corresponding agent-based representations; the Specialists can perform this task independently. Nonetheless, extra visibility via the Store can increase the utilization of resources, motivating the providers to provide the corresponding agent representations directly.

Given the central placement of agent-based modeling in the Store, we proceed to provide the necessary background in the next Section.

# Multi-Agent Systems

The agent-based modeling is an approach for representing and managing an arbitrary set of resources in a natural and intuitive manner. The term resource is generic and can cover any kind of physical or logical, distributed or local, static or dynamic potential . Diverse examples include physical processors, link capacity monitors and data flow routing services within a computer network.

The management of resources commonly presents cross-cutting concerns: one management aspect may exemplary refer to the energy consumption of a physical computer network, while another may refer to the efficiency of the logical path allocation services that run within it. In agent-based modeling, such concerns are denoted as Environments, as shown in Fig. . Each environment acts as a unified context for autonomous software processes, the Agents.

An agent represents and manages one or more resources within an environment. For instance, an agent may represent and manage the routing table of a single router, or the peering relations of a complete autonomous system, depending on the required level of abstraction. An agent manages its associated resource autonomously, i.e., without the need for constant human supervision, towards a clearly defined Objective. Returning to the routing table example, an agent's objective can be to find and maintain a list of the shortest network paths within a hybrid network comprising SDN and legacy parts. In order to accomplish their objective, agents commonly collaborate with each other and react to global changes in their environment. Additionally, they can move within their environment to optimize their objective: An agent representing a virtual machine can relocate to another physical site that offers better performance.

The agent-based approach presents enticing features that constitute it an ideal platform for implementing the Socket Store. From a practical point of view, agents and environments are programmatically described by customizable object classes, which are the norm in software development. Moreover, existing agent development frameworks provide: i) Agent management facilities comprising objective definition, instantiation, inter-agent communication, movement and destruction. ii) Environment synchronization, allowing agents to obtain a central and coherent view of their environment, even when the represented resources are purely distributed. This generality constitutes multi-agent systems capable of efficiently representing both distributed *and* centralized network architectures, such as SDNs. iii) GUIs for the visualization of the inspection of agents and their environments. iv) Cross-platform operation regarding programming languages for describing the agents and the hardware/software platforms they can reside in. Additional benefits include massive scalability and strong community support .

# The Socket Store Architecture

The architecture of the Socket Store comprises (Fig. ): (i) A front-end and a back-end interface, for Developers and Specialists respectively, (ii) The Agent Type Library, (iii) The Module Repository, (iv) The Store Access Control, (v) The Agent Instance Management Platform, operating in tandem with a Distributed Agent Communication platform. Finally, closely related but external to the Store are the Testbeds, which allow for automated module validity check and evaluation when present.

The Specialists employ the Store's back-end to submit, review, revise and maintain modules. A module is an XML description of a set of agents which are created, placed, moved, monitored and destroyed per the instructions, i.e., the Network Side Directives (NSD) of the Specialist authoring the module. A module is associated with a performance Metric, which expresses its promised objective. The agent types and the metrics are organized as the Agent Type Library of the Store. Within the Library, the agent types are further classified as Resource Agents, which directly model a network resource, or as Adapters, which combine other agents to provide a different modeling approach or functionality.

A module also contains a Device-Side Agent (DSA), an *open-source*, lightweight agent intended to be executed at the end-user device, and perform access control to the Store and, subsequently, to network resources. The Store provides standard DSAs for different end-device operating systems. An important but optional capability, is that specialists can extend these standard DSAs to offer advanced interaction between the application and the network resources. An example is given in Section . Modules accepted for publication at the Store have their descriptions and DSAs stored centrally at the Module Repository.

The Developers use the Store's front-end interface to search for and purchase modules fitting the requirements of their applications. The front-end is intended to offer a smart search engine, a module ranking and a commenting system. Moreover, each module can be accompanied by its associated Metric values, logged during its operation. Thus, an objective module ranking can be attained apart from a community-based one. Once a module has been obtained, the Developer incorporates the DSA to his application, with trivial coding burden.

An end-user that executes the application on his device will trigger the execution of the DSA. The DSA then undergoes authorization by the Store Access Control, described later in Section , ensuring that the application has indeed access to execute the associated NSD of the module. Upon successful authorization, the Agent Instance Management Platform executes the NSD creating the respective agents within their environments, which in turn allocate the corresponding network resources. While the module is still in review phase, the NSD is executed within testbeds, providing automated module validity check and performance evaluation. Published modules are ready for deployment in their intended production networks, which can include the Internet.

The Distributed Agent Communication platform supports the Agent Instance Management, providing robust inter-agent and agent-Store communication. Existing distributed messaging solutions, such as ZeroMQ , fill in this role. The Agent Instance Management is thus enabled to provide a central view of the active agents and their resources, regardless of the actual distribution degree of the network. Thus, the Store attains a unified modeling and management of both completely centralized network technologies, such as SDNs, or completely decentralized. The centralization benefits the Specialists and the Resource Providers: it can facilitate the development and debugging of modules, as well as the optimization of network resource allocation.

## Security and Trust Model

The Socket Store aligns itself with existing, deployed and well-received security and trust approaches, without introducing Store-specific novelties. Particularly, it leverages existing application distribution channels, resource slicing and control/data Separation as follows:

Socket Store modules are distributed as parts of standard applications and not in a stand-alone fashion. Thus, they employ the same privacy agreements and trust levels of the corresponding application distribution channels. Distribution via App Stores also solves the problem of access control. Upon DSA invocation, the Socket Store contacts the App Store to verify ownership of the *application* containing the module. The same, existing approach is employed when modules are provided as in-app purchases. Distribution outside App Stores relies on any standard licensing approach, such as existing license server schemes. Notice that the DSAs are supplied as open-source in any case, to ensure the Developer's trust.

Once the NSD part of a module is executed, the instantiated agents call upon existing ways of interacting with resources offered by the Providers . This approach does not upset the existing trust model associated with the use of such resources. Additionally, existing sand-boxing techniques, such as network resource slicing remain active and unobstructed by the Store, ensuring that programming faults or malevolent behavior does not affect the network as a whole.

A key-point in the Socket Store trust model is that it operates at the network control plane. Via their modules, Specialists propose optimizations in the management of resources, without access to the actual data flowing through the network. As a step against indirect security breaches, all agent definitions are required to rely on the Store Agent Library only, and not on custom libraries.

The Store logs all agent actions to trace errors or malevolent behavior. Moreover, all module submissions are made by eponymous Specialists, who are encouraged to extend the prestige of their professional profile within the Store. This approach entails direct liability of actions, which can act as another deterring factor for malevolent behavior.

Finally, Store failures are never critical: DSAs resort to standard socket connectivity in case of any unrecoverable error at either the device or the Store-side.

## Operational Expenditure Model

The Store does not impose restrictions to the pricing of modules, which is left to the discretion of their creators, following other well-known store paradigms. Nonetheless, the Store offers a standard way to keep track of the operational resource expenditure per module, facilitating their pricing.

Certain network resources have a running costs, defined by the Providers. These can exemplary refer to expended storage, processing time (e.g., for involved VMs), served traffic volume or even paid access to external services, such as Internet monitoring and analysis databases. A module instance is registered as a user of such resources, for each separate Provider it employs, using the corresponding Provider-offered interfaces. The Specialist authoring the module is required to implement a standard member function, $\texttt{.cost()}$, which should iterate over the underlying Provider interfaces, summarize and return the running resource expenditure up to the function callback. Weight functions can then be freely applied to the reported resource expenditure, obtaining the total monetary cost.

This approach allows for the dynamic enabling of modules, based on the user's preferences. For example, a module of robust connectivity can be enabled via the app only when exchanging sensitive data. Overlay mirror-paths can then be setup and charged only for the time needed. Actual payment can be delegated fully to the end-user, without the Store's mediation in the transaction.

# Implementation

A Socket Store prototype has been implemented for the purpose of concept evaluation. The prototype incorporates the components detailed in Section , within the AnyLogic platform . AnyLogic is a JAVA-based multi-paradigm software development environment, which facilitates agent-based development. Metrics, Facilities and Resource/Adapter agents are organized in the platform's palettes, and the Specialist is offered a GUI enabling: i) the composition of complex modules, ii) their visualization, iii) their debugging and run-time management. A central view of instantiated agents and their environments is readily provided, employing the Agent Management tools offered by AnyLogic. NSDs are stored in XML format, and can be executed as such by the Store. DSAs are exported in JAVA $\texttt{.jar}$ portable format.

The considered network is an SDN environment emulated in MiniNet, controlled by the POX controller . This is directly mapped to an agent environment, with Resource Agent types representing:

-   An OpenFlow switch, allowing for inspectable and editable routing tables.

-   A network link providing read-only access to static (end-points and capacity) as well as dynamically monitored attributes (packet latency, transfer rate and load).

Resource Agent instances are organized in iterable JAVA collections by the AnyLogic platform. Moreover, the contained agent management provides a centralized, graphical representation of all agent instances, readily yielding the network topology. Thus, the specifics of the underlying SDN technology are abstracted. The same agent representation and logic can be ported, e.g., to the Internet, using globally distributed VMs that form overlay networks.

An Adapter Agent is introduced for this evaluation, coupled with an example of extended DSA. The K-paths Mirroring Agent (KM) duplicates a connection and its data over K link-disjoint paths, with approximately identical attributes. It receives as inputs two host endpoints (IPs and ports), the integer K and the intended data transfer rate and latency. A DSA invokes the KM agents passing these arguments. Subsequently, KM is instantiated and attempts to allocate the K mirror-paths between the endpoints. If the allocation fails, the DSA is informed and a regular socket is opened. The DSA may also relay the failure event to the application business logic to initiate a negotiation instead. In the latter case, the developer should handle the negotiation logic as well.

We proceed to present the *flash-delivery* module, which employs the KM Agent. The module usage scenario corresponds to critical data delivery between two mobile devices over the Internet. Each packet from a sending *Device A* must reach the recipient *Device B* with high probability (i.e., using packet replication over link-disjoint paths) and within a strict deadline (set to $5\,\text{ms}$), without ACKs or re-transmissions on errors. A single packet loss or deadline expiry corresponds to failed communication. Flash-trades, critical VM migrations and RPC signaling are some potential applications .

The emulated setup is given in Fig. . Each link has an effective capacity of $100\,\text{Mbps}$ and $0.5$ ms latency. We note that the topology comprises $5$ routers due to limitations of the emulation environment. Specifically, the accuracy of latency measurements in MiniNet can be low for large topologies .

The network is monitored and controlled by the POX controller located at node $\#3$, which can deploy/retract paths per flow and maintain a consistent network-wide state (topology, nodes, link latency).

Each device has two separate physical network cards, and a DSA to communicate with the Store. At the DSA of Device B, a developer calls a single line of code, i.e. the exposed $\texttt{bind()}$ function, which periodically notifies the Store of the current connectivity details of Device B. At the DSA of the sending side (Device A), a developer calls a single function, i.e., $\texttt{connect("Device\_B")}$. The alias $\texttt{"Device\_B"}$ is automatically resolved by the DSA to the actual connectivity details of B (available at the Store). Subsequently, the KM agent is instantiated and communicates with the POX controller on the application's behalf, setting up $2$ mirrored, link-disjoint paths. (Notice that the default paths may not be link-disjoint, as shown in Fig. ). Finally, the data is sent on both paths, and device B discards already received packets.

The IPFW tool is used for introducing a sharp latency increase at link $4\to B$  during the critical data transmission, as shown in Fig. . Naturally, when the module is activated, the data transmission remains successful.

# Discussion and Outlook

The Store offers a novel way for modeling and understanding the interactions among the various network processes and the applications. Using the mature, multi-agent approach, the Store promotes the interconnectivity and re-usability of network modules. Moreover, given that the Store modules carry a clear objective, the Resource Providers can obtain a better understanding of the developer's needs, evolving their offered services accordingly. In this aspect, note that evolving the Internet protocols and infrastructure presently constitutes a significant research goal, especially towards improved QoS . Such paradigms are top-to-bottom approaches for advancing the Internet: changes to the Internet core are required, without assuring a monetary profit for the Network Resource providers first. The Socket Store works from the complimentary, bottom-to-top direction. It can create a client base to motivate this Internet core evolution.

Deriving from its compatibility with the current Internet operation, the Store is not a form of paid prioritization. The Store is a clear case of outsourced programming: Specialists handle the creation of modules for a price. This can translate to better Internet service compared to applications outside the Store. However, this performance boost will be owed to "smarter" network programming, not to unfair access provision. Additionally, the Store is not a replacement for existing client-side code libraries, given that basic connectivity can be sufficient for many applications. Finally, the Store does not claim an exclusiveness over outsourcing. Freelance Specialists already offer their services and can freely continue to do so outside any Store. However, the highly successful cases of mobile app stores have shown that the added visibility benefits the Specialists. The Socket Store has the additional prestige potential stemming from its academic research orientation.

The Store is intended as a unified platform for hosting modular solutions for a wide variety of application contexts, beyond the described functionality on the Internet and access networks. For instance, the Store is the intended interface for an emerging hardware paradigm, the software-defined metasurfaces (SDMs) . SDMs are a novel class of materials with customizable electromagnetic behavior. They allow for custom steering or absorption of impinging electromagnetic waves, enabling novel applications in high-resolution imaging and networking. Modular solutions that describe intended behaviors of SDMs can be expressed as Store modules, and be activated upon demand by DSAs. Creating, chaining and multiplexing such functionalities is a future research goal.

# Conclusion

The present work introduced the Socket Store, a novel approach for realizing network-aware applications and application-aware networks. The Store makes state-of-the-art research knowledge accessible to mobile/desktop software developers. It motivates appropriately trained network specialists to publish their work in the form of reusable software modules. Network providers are encouraged to expose their infrastructure capabilities as blocks for building advanced modules. Developers purchase access to a module and enjoy state-of-the-art networking performance, while minimizing network development time and its associated expenses. The Socket Store acts as a focal point for close collaboration among network providers, researchers and developers, with distinct, fitting roles and benefits.

[^1]: This work was funded by the European Union via the Horizon 2020: Future Emerging Topics call (FETOPEN), grant EU736876, project VISORSURF (http://www.visorsurf.eu).


author: Alex Tamkin[^1]; Miles Brundage$^{*}$; Jack Clark[^2]; Deep Ganguli
title: Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models

# Introduction

On October 14th, 2020, researchers from OpenAI, the Stanford Institute for Human-Centered Artificial Intelligence, and other universities convened to discuss open research questions surrounding [GPT-3](https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html), the largest publicly-disclosed dense language model at the time.

The meeting took place under Chatham House Rules. Discussants came from a variety of research backgrounds including computer science, linguistics, philosophy, political science, communications, cyber policy, and more. Broadly, the discussion centered around two main questions:

1.  **What are the technical capabilities and limitations of large language models?** The discussion touched on several key areas including: the surprising impact of scale on model capabilities, the difficulty in assessing whether large language models truly understand language, the importance of training models on multiple data modalities, and challenges in aligning model objectives with human values.

2.  **What are the societal effects of widespread use of large language models?** The discussion touched on several key areas including: difficulties in scoping all possible uses (or misuses) of general purpose language models, challenges organizations may face in model deployment, the potential for these models to algorithmically spread disinformation, difficulties in mitigating model bias (e.g., racial, gender, religious, etc.), and the impact of language model-based automation on the labor market.

While the conversation was collegial and productive, there was a sense of urgency to make progress sooner than later in answering these questions. Here, we provide a detailed summary of the discussion organized by the two themes above.[^3] We conclude with a list of potential future research directions inspired by the discussion.

# Technical Capabilities and Limitations

## Scale

GPT-3 is one of the largest publicly-disclosed language models — it has 175 billion parameters and was trained on 570 gigabytes of text. For comparison, its predecessor, GPT-2 (which is functionally similar to GPT-3) has 1.5 billion parameters and was trained on 40 gigabytes of text. While GPT-2 displayed some zero-shot generalization to downstream tasks, GPT-3 further displayed the ability to learn more novel tasks when given examples in context. Participants found it remarkable that such capabilities emerge merely from scaling model and training data size.

One person remarked that the growth in model capabilities as they scale "feels like a law of physics or thermodynamics" in its [stability and predictability](https://arxiv.org/abs/2001.08361). Several participants were optimistic that these trends would continue even for models much larger than GPT-3, yielding ever-stronger models capable of more advanced few-shot learning of new skills from a small number of training examples.

One participant remarked that the scale of models like GPT-3 was reminiscent of large particle accelerator experiments, which require many people with diverse backgrounds to execute. For example, when training such large models, different teams with diverse expertise must collaborate to run experiments, build and maintain the computing infrastructure, develop the algorithms, and continuously interrogate the model's capabilities for possible problems (e.g., bias, misuse, safety concerns, etc.). The latter point is referred to as "red-teaming" throughout the rest of this document.

## Understanding

What constitutes "understanding" in a language model, and does GPT-3 fulfill this definition? Some leaned towards definitions based on strong notions of intelligence, which require models to possess intentionality or the ability to respond to requests in the real world. Others suggested that there were even weaker notions of intelligence that models had yet to satisfy, including robustness to adversarial examples — data examples that easily confuse an AI system but not humans. Participants suggested that getting things "mostly right" may not be sufficient for understanding if the model performs poorly on rare but important inputs.

Another definition of understanding centered around the notion of causality, in that models that truly understand should grasp the causal relationship between features of the data and the desired behavior. Some argued that language models were destined to exploit "spurious correlations" or "shortcut features" inherent in the data, and thus lacked a true underlying causal model. However, one participant suggested a different view — that with enough data, language models could encounter "natural experiments" that could enable the model to learn causal relationships from observational data in a similar manner as human economists often do in their research.

Some participants argued against binary thresholds for understanding, recalling that children and adults gradually acquire greater mastery over time. For example, one participant quoted a prominent physicist who quipped that he only understood thermodynamics the third time he taught it. Another participant pushed back against singular notions of understanding, noting debates between linguists and philosophers about whether meaning is derived from the relationship of expressions to each other or to some external ground truth.

Finally, some participants offered resistance to the focus on understanding, arguing that humans are able to accomplish many tasks with mediocre or even poor understanding, including a non-French speaker who recently won the French Scrabble championships. Some gently suggested that perhaps a judgment about whether GPT-3 understands language in the relevant way is irrelevant to successful performance of tasks.

In a memorable line, one participant also remarked on the inverse problem of humans' ability to understand large language models: "GPT-3 is completely alien…it's the first thing I've seen where it's not a dumb thing to ask whether it's AGI." Here, AGI refers to Artificial General Intelligence, or the ability of a machine to learn and understand anything a human can.

## Multimodality

Much of the conversation considered the importance of multimodal models — language models trained on text and data from other modalities, e.g., images, audio recordings, etc. Participants largely agreed in their predictions that large multimodal models will become more prevalent and enable more diverse capabilities.[^4] However, some argued that GPT-3 is already trained on multimodal data, in that the training data contains prose, structured data tables, and computer code. Others suggested that the main benefit of multimodal training might be to improve the speed at which models acquire useful capabilities, as the interaction between different data modalities may provide a stronger learning signal than each data modality in isolation provides. Finally, some commented that no single additional modality was critical to language use, given that humans differ in the range of sensory modalities they have access to.

## Alignment

Participants discussed the need to better align model objectives with human values. For example, one participant mentioned some language models treat all symbols (e.g., nouns, prepositions, numbers, etc.) equally, but humans care much more about, for example, incorrectly stating someone's age than about misplacing a preposition. Several other participants emphasized the importance and challenge of better optimizing for factual accuracy and robustness to adversarial examples. Aligning human and model objectives was seen to be especially important for "embodied" AI agents which learn through active interaction with their environment. Discussants emphasized the dual importance of developing better algorithms for "steering" agents towards human values, as well as fostering cross-disciplinary collaborations to better clarify what "human values" means, especially given diversity across individuals and communities and the prevalence of bias in available datasets.

# Effects of Widespread Use

## Capabilities

GPT-3 has an unusually large set of capabilities, including text summarization, chatbot behavior, search, code generation, and essay generation. One discussant stated that such a large "capability surface" makes it challenging to both scope the full array of uses (because GPT-3 can take in arbitrary inputs, it is *a priori* impossible to anticipate all potential behaviors of the model) and to ensure their safety to people and societies. Participants noted that, by putting GPT-3 behind a controlled-access API, OpenAI is able to constrain the model's use more easily than if they open sourced it. However, open questions remain. For example, who gets access and why? How can one provide model access to support a large community to red-team (interrogate the model for potential misuse and develop mitigation strategies) at scale?

## Deployment

Participants discussed several options for defining and addressing the ethical and societal challenges of deploying large language models. One suggestion was to increase the computing resources available to academia so that it would be easier for academics to do research that informs the deployment of large language models. Someone suggested that laws requiring disclosure of when AI is being used to generate text could be helpful in managing the effects of large language models. Another participant asked what metrics might be used to evaluate whether language models are having a societally beneficial effect, and there was general agreement that this is a challenging but important task.

Several participants noted that OpenAI and other organizations will not have a monopoly on large language models forever. Participants suggested that developers may only have a six- to nine-month advantage until others can reproduce their results. It was widely agreed upon that those on the cutting edge should use their position on the frontier to responsibly set norms in the emerging field. Additionally, some participants pointed out that, due to standard advances in technology, it will only become easier for other actors to replicate models like GPT-3 over time. This further suggests the urgency of using the current time window, during which few actors possess very large language models, to develop appropriate norms and principles for others to follow.

## Disinformation

A major discussion point considered the deliberate misuse of language models for purposes such as generating disinformation. More specifically, models like GPT-3 can be used to create false, misleading, or propagandistic essays, tweets, and news stories *de novo*. One participant was skeptical about the magnitude of these likely risks since many previous technologies (e.g. photography and Photoshop) sparked similar concerns and have already raised societal awareness of the risks of disinformation. Furthermore, while automated generation of disinformation may be feasible in principle, human labor may still be more cost-effective for such purposes. Others disagreed, and saw automated generation as much more cost-effective than training and paying humans to generate disinformation. Participants agreed that empirically investigating the economics of automated vs human generated disinformation is important.

Thinking ahead, someone suggested considering a future in which language models can generate text that is not just coherent on commonly discussed topics, but highly persuasive on arbitrary topics. Another participant suggested that GPT-3 or other future language models could make disinformation hard or impossible to detect at the level of content, forcing reliance on metadata by online platforms. Relatedly, someone suggested that the existence of systems like GPT-3 should spur more use of cryptography to authenticate media.

## Bias

GPT-3 exhibits several racial, gender, and religious biases. One discussant analogized the difficulty of addressing language model bias to the problem of content moderation on online platforms — despite the difficult normative issues in both cases, there are still some areas of relative consensus and opportunities for mitigation. For example, online platforms agree on the need to address child pornography or egregious threats of violence, and the concept of "protected classes" in discrimination law provides a useful initial framework for thinking about some language model biases.

Several workshop participants noted that it is difficult to define what it means to mitigate bias in large language models in a universal manner, since appropriate language use is highly contextual. One participant noted that all datasets are biased in some ways, so the challenge is not eliminating all bias but addressing harmful biases according to some set of normative and/or legal criteria. Some suggested that companies like OpenAI do not have the appropriate standing and should not aim to make such decisions on behalf of society. Someone else observed that it is especially difficult to think about mitigating bias for multi-purpose systems like GPT-3 via changes to their training data, since bias is typically analyzed in the context of a particular use cases.

Participants discussed a wide variety of possible means of addressing harmful biases in language models, including:

-   Changes to the initial training data to mitigate bias *a priori*

-   Training a separate model to filter content generated by a language model

-   Fine-tuning a large language model on data with desired properties

-   Tagging data so that the model learns to distinguish among certain forms of content (see e.g. [CTRL](https://arxiv.org/abs/1909.05858))

-   Training models to be more ["fact-aware"](https://arxiv.org/abs/1906.07241)

-   Reinforcement learning with [human feedback](https://openai.com/blog/learning-to-summarize-with-human-feedback/)

-   Leveraging the model's own knowledge to improve outputs (e.g., with careful prompt design)

-   Developing more expansive suites of "bias tests" that models can be run through prior to deployment

-   Red-teaming the model at scale by engaging trusted partners to work with the model and through limited commercial offerings.

None of these approaches was considered a panacea. For example, steering a model with human feedback still raises the question of who the human labelers are or how they should be chosen, and content filters can sometimes undermine the agency of the very groups that they are intended to protect (e.g., marginalized groups reclaiming words or phrases that are used as slurs by majority groups). One participant argued that keeping a human in the loop of text generation is critical for addressing these issues. Some participants emphasized that certain use cases should be avoided given the limitations of existing techniques, and that text generation applications vary widely in terms of open-endedness and risk. For example, detecting regular expressions is much more tractable to do safely than managing a suicide hotline.

## Economy

Another theme of the discussion considered the economic implications of models like GPT-3. Participants observed that current jobs that involve reading or analyzing text vary widely in their desirability, with some being more enjoyable (e.g., creative writing or reading and summarizing reports) and others often being traumatizing or alienating (e.g., content moderation). This raises the question of when jobs, or what kinds of jobs, should or shouldn't be automated by large language models. One participant suggested that leaving such decisions up to companies would likely have adverse consequences. Education was also mentioned as a societal area likely to be affected by large language models, via changes to the essay writing process as well as evaluation of text. One participant pointed out that providing API access to a variety of groups from different sectors of society can help provide an early signal of potential societal changes.

# Future Research Directions

The following research questions were inspired by the discussion:

-   Can we better understand why language models improve so much with scale? Can this enable us to build models which scale more efficiently?

-   What are the limits of scaling? Will scale lead to strong causal reasoning, symbolic manipulation, commonsense understanding, and robustness to a wider class of inputs? Or will different techniques be necessary?

-   How can we understand the limits of what large language models are capable of? Can we enable models to ask for help or clarification, or abstain when they are unsure?

-   How can we develop new neural network architectures and algorithms that enable efficient learning from diverse, multimodal data beyond text?

-   What are the opportunities and tradeoffs involved in different approaches to steering the outputs of large-scale language models to be more aligned with human values?

-   How should access to models like GPT-3 be allocated, balancing considerations like security, replicability, and fairness? What kinds of tests do we need to develop in order to qualify language models like GPT-3 as being safe or unsafe for use in particular contexts?

-   What can academia do to best position itself to develop guardrails for the industrial development of such models - including advocating for sufficient funding to replicate the compute resources required to train them?

-   How can we best foster cross-disciplinary collaboration to [understand](https://arxiv.org/abs/1803.09010) and manage the biases in large datasets and model representations of such datasets?

-   How can we best characterize the potential "threat landscape" for such models; e.g., do we need to spend more time worrying about how models like this could be used by profit-driven actors to generate lots of low-grade spam, or should we be more worried about state-based actors using models to generate persuasive text for use in disinformation campaigns?

-   How cost-effective and skill-intensive would it be for malicious actors to misuse language models for various purposes, compared to alternative methods of achieving the same goals?

[^1]: Equal contribution

[^2]: Work carried out while employed at OpenAI

[^3]: Since this is a summary of discussions, rather than a research paper, we do not include references. Rather, we hyperlink to relevant papers that were discussed at the workshop. For a more comprehensive set of references related to some of these issues, we point readers to the original [GPT-3 paper](https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html) and to recent work of [Bender and Gebru et al](https://faculty.washington.edu/ebender/papers/Stochastic_Parrots.pdf) published a few months after this workshop.

[^4]: In fact, shortly after the workshop, OpenAI released [DALL-E](https://openai.com/blog/dall-e/), which is a multimodal version of GPT-3 trained on both images and text.


abstract: These instructions provide guidelines for preparing papers for IEEE Transactions, but this version is specifically written to describe submission to IEEE TMI. Use this document as a template if you are using LaTeX. Otherwise, use this document as an instruction set. The electronic file of your paper will be formatted further at IEEE. Paper titles should be written in uppercase and lowercase letters, not all uppercase. Avoid writing long formulas with subscripts in the title; short formulas that identify the elements are fine (e.g., "Nd–Fe–B"). Keep the title short and do not write "(Invited)" in the title. Full names of authors are preferred in the author field, but are not required. Put a space between authors' initials. Only authors may appear in the author line of a manuscript. Authors are defined as individuals who have made an identifiable intellectual contribution to a manuscript to the extent that the individual can defend its contents. Define all symbols used in the abstract. Do not cite references in the abstract. Keep the abstract to 250 words or less.
author: First A. Author, , Second B. Author,and Third C. Author, Jr.,
[^1] [^2] [^3] [^4]
title: Preparation of Papers for IEEE TRANSACTIONS ON MEDICAL IMAGING

# Introduction

document is a template for LaTeX. You are encouraged to use it to prepare your manuscript. If you are reading a paper or PDF version of this document, please download the LaTeX.zip file from the IEEE Web site at https://www.embs.org/tmi/authors-instructions/ to prepare your manuscript. You can also explore using the Overleaf editor at https://www.overleaf.com/blog/278-how-to-use-overleaf-with-ieee-collabratec-your-quick-guide-to-getting-started#.xsVp6tpPkrKM9

# Guidelines for Manuscript Preparation

Do not change the template font sizes or line spacing to squeeze more text into a limited number of pages. The preferred font is 10-pt Times New Roman. Use italics for emphasis; do not underline words.

Place your figures in the text as you expect them to appear in print. Further instructions on figure usage appear in Section VI. Although IEEE will do the final formatting of your paper, we expect you to approximate the final form appearance for all versions submitted to TMI via ScholarOneto the extent possible.

## Abbreviations and Acronyms

Define abbreviations and acronyms the first time they are used in the text, even after they have already been defined in the abstract. Abbreviations such as IEEE, SI, ac, and dc do not have to be defined. Abbreviations that incorporate periods should not have spaces: write "C.N.R.S.," not "C. N. R. S." Do not use abbreviations in the title unless they are unavoidable (for example, "IEEE" in the title of this article).

## Other Recommendations

Use one space after periods and colons. Hyphenate complex modifiers: "zero-field-cooled magnetization." Avoid dangling participles, such as, "Using , the potential was calculated." It is not clear who or what used . Write instead, "The potential was calculated by using ," or "Using , we calculated the potential."

Use a zero before decimal points: "0.25," not ".25." Use "cm$^{3}$," not "cc." Indicate sample dimensions as "0.1 cm $\times$ 0.2 cm," not "0.1 $\times$ 0.2 cm$^{2}$." The abbreviation for "seconds" is "s," not "sec." Use "Wb/m$^{2}$" or "webers per square meter," not "webers/m$^{2}$." When expressing a range of values, write "7 to 9" or "7–9," not "7$\sim$`<!-- -->`{=html}9."

A parenthetical statement at the end of a sentence is punctuated outside of the closing parenthesis (like this). (A parenthetical sentence is punctuated within the parentheses.) In American English, periods and commas are located within quotation marks, like "this period." Other punctuation is placed "outside"! Avoid contractions; for example, write "do not" instead of "don't." The serial comma is preferred: "A, B, and C" instead of "A, B and C."

If you wish, you may write in the first person singular or plural form using the active voice ("I observed that $\ldots$" or "We observed that $\ldots$" instead of "It was observed that $\ldots$"). Remember to check spelling. If your native language is not English, please have a native English-speaking colleague to carefully proofread your paper.

# Math

## Equations

Number equations consecutively with equation numbers in parentheses flush with the right margin, as appears in . Refer to "," not "Eq. " or "equation ," except at the beginning of a sentence: "Equation is $\ldots$ ." To make your equations more compact, you may use the solidus ( / ), the exp function, or appropriate exponents. Use parentheses to avoid ambiguities in denominators. Punctuate equations when they are part of a sentence, as in $$E=mc^2.\label{eq}$$

Be sure to define the symbols in your equation before the equation appears or immediately following. Italicize symbols ($T$ might refer to temperature, but T is the unit tesla).

## LaTeX-Specific Advice

Use "soft" (e.g., `\eqref{Eq}`) cross references instead of "hard" references (e.g., `(1)`). This will make it possible to combine sections, add equations, or change the order of figures or citations without having to manually change equation references.

Do not use the `{eqnarray}` equation environment. Use `{align}` or `{IEEEeqnarray}` instead. The `{eqnarray}` environment leaves unsightly spaces around relation symbols.

Note that the `{subequations}` environment in LaTeX will increment the main equation counter even when there are no equation numbers displayed.

B-.05emi-.025em b-.08em T-.1667em.7ex-.125emX only functions in conjunction with local .bib files. If you use B-.05emi-.025em b-.08em T-.1667em.7ex-.125emX to produce the bibliography you must attach the .bib files.

LaTeX can't read your mind. If you assign the same label to both a subsubsection and a table, you may find that Table I has been cross referenced as Table IV-B3.

LaTeX does not have precognitive abilities. If you put a `\label` command before the command that updates the counter it's supposed to be using, the label will pick up the last counter to be cross referenced instead. In particular, a `\label` command should not go before the caption of a figure or a table.

Do not use `\nonumber` inside the `{array}` environment. It will not stop equation numbers inside `{array}` and it might stop a wanted equation number in the surrounding equation.

If you are submitting your paper to a colorized journal, you can use the following two lines at the start of the article to ensure its appearance resembles the final copy:

```latex
\begin{small}
\begin{tabular}{l}
\verb+\++\texttt{documentclass[journal,twoside,web]\{ieeecolor\}}\\
\verb+\++\texttt{usepackage\{\textit{Journal\_Name}\}}
\end{tabular}
\end{small}
```

# Units

Use either SI (MKS) or CGS as primary units. (SI units are strongly encouraged.) English units may be used as secondary units (in parentheses). For example, write "1 kg (2.2lb)." An exception exists for when English units are used as identifiers in commercial products, such as a "3-in disk drive." Avoid combining SI and CGS units, such as current in amperes and magnetic field in oersteds. This often leads to confusion because equations do not balance dimensionally. If you must use mixed units, clearly state the units for each quantity in an equation.

The SI unit for magnetic field strength $H$ is A/m. However, if you wish to use units of T, either refer to magnetic flux density $B$ or magnetic field strength symbolized as $\mu _{0}H$. Use the center dot to separate compound units, e.g., "A$\cdot$m$^{2}$."

# Guidelines for Graphics Preparation and Submission

## Types of Graphics

The following list outlines the different types of graphics published in IEEE journals. They are categorized based on their construction, and use of color / shades of gray:

### Color/Grayscale figures

Figures that are meant to appear in color, or shades of black/gray. Such figures may include photographs, illustrations, multicolor graphs, and flowcharts.

### Line Art figures

Figures that are composed of only black lines and shapes. These figures should have no shades or half-tones of gray, only black and white.

### Author photos

Not allowed for papers in TMI.

### Tables

Data charts which are typically black and white, but sometimes include color.

Units for Magnetic Properties

Symbol
Quantity

Conversion from Gaussian and

CGS EMU to SI a

Φ
magnetic flux
1 Mx  → 10−8 Wb  = 10−8 V⋅s

B

magnetic flux density,

magnetic induction

1 G  → 10−4 T  = 10−4 Wb/m2

H
magnetic field strength
1 Oe  → 103/(4π) A/m

m
magnetic moment

1 erg/G = 1 emu

 → 10−3 A⋅m2 = 10−3 J/T

M
magnetization

1 erg/(G⋅cm3)= 1 emu/cm3

 → 103 A/m

4πM
magnetization
1 G  → 103/(4π) A/m

σ
specific magnetization
1 erg/(G⋅g) = 1 emu/g → 1 A⋅m2/kg

j

magnetic dipole

moment

1 erg/G = 1 emu

 → 4π × 10−10 Wb⋅m

J
magnetic polarization

1 erg/(G⋅cm3)= 1 emu/cm3

 → 4π × 10−4 T

χ, κ
susceptibility
1  → 4π

χρ
mass susceptibility
1 cm3/g  → 4π × 10−3 m3/kg

μ
permeability

1  → 4π × 10−7 H/m

 = 4π × 10−7 Wb/(A⋅m)

μr
relative permeability
μ → μr

w, W
energy density
1 erg/cm3 → 10−1 J/m3

N, D
demagnetizing factor
1  → 1/(4π)

Vertical lines are optional in tables. Statements that serve as captions for the entire table do not need footnote letters.

aGaussian units are the same as cg emu for magnetostatics; Mx = maxwell, G = gauss, Oe = oersted; Wb = weber, V = volt, s = second, T = tesla, m = meter, A = ampere, J = joule, kg = kilogram, H = henry.

## Multipart figures

Multipart figures are comprised of more than one sub-figure presented together. If a multipart figure is made up of multiple figure types (one part is lineart, and another is grayscale or color) the figure should meet the strictest applicable guidelines.

## File Formats For Graphics

Format and save your graphics as one of the following approved file types: PostScript (.PS), Encapsulated PostScript (.EPS), Tagged Image File Format (.TIFF), Portable Document Format (.PDF), Portable Network Graphics (.PNG), or Metapost (.MPS). After the paper is accepted, any included graphics must be submitted alongside the final manuscript files.

## Sizing of Graphics

Most charts, graphs, and tables are one column wide (3.5 inches / 88 millimeters) or page wide (7.16 inches / 181 millimeters). The maximum depth of a graphic is 8.5 inches (216 millimeters). When choosing the depth of a graphic, please allow space for a caption. Authors are allowed to size figures between column and page widths, but it is recommended not to size figures less than column width unless necessary.

## Resolution

The proper resolution of your figures will depend on the type of figure it is as defined in the "Types of Figures" section. Author photographs, color, and grayscale figures should be at least 300dpi. Lineart, including tables should be a minimum of 600dpi.

## Vector Art

While IEEE does accept and even recommends that authors submit artwork in vector format, it is our policy is to rasterize all figures for publication. This is done in order to preserve figures' integrity across multiple computer platforms.

## Colorspace

The term colorspace refers to the entire sum of colors that can be represented within a given medium. For our purposes, the three main colorspaces are grayscale, RGB (red/green/blue) and CMYK (cyan/magenta/yellow/black). RGB is generally used with on-screen graphics, whereas CMYK is used for printing purposes.

All color figures should be generated in RGB or CMYK colorspace. Grayscale images should be submitted in grayscale colorspace. Line art may be provided in grayscale OR bitmap colorspace. Note that "bitmap colorspace" and "bitmap file format" are not the same thing. When bitmap colorspace is selected, .TIF/.TIFF are the recommended file formats.

## Accepted Fonts Within Figures

When preparing your graphics IEEE suggests that you use of one of the following Open Type fonts: Times New Roman, Helvetica, Arial, Cambria, and Symbol. If you are supplying EPS, PS, or PDF files all fonts must be embedded. Some fonts may only be native to your operating system; without the fonts embedded, parts of the graphic may be distorted or missing.

A safe option when finalizing your figures is to strip out the fonts before you save the files, creating "outline" type. This converts fonts to artwork that will appear uniformly on any screen.

## Using Labels Within Figures

### Figure Axis labels

Figure axis labels are often a source of confusion. Use words rather than symbols. As an example, write the quantity "Magnetization," or "Magnetization M," not just "M." Put units in parentheses. Do not label axes only with units. As in Fig. 1, for example, write "Magnetization (A/m)" or "Magnetization (A$\cdot$m$^{-1}$)," not just "A/m." Do not label axes with a ratio of quantities and units. For example, write "Temperature (K)," not "Temperature/K."

Multipliers can be especially confusing. Write "Magnetization (kA/m)" or "Magnetization (10$^{3}$ A/m)." Do not write "Magnetization (A/m)$\,\times\,$`<!-- -->`{=html}1000" because the reader would not know whether the top axis label in Fig. 1 meant 16000 A/m or 0.016 A/m. Figure labels should be legible, approximately 8 to 10 point type.

### Subfigure Labels in Multipart Figures and Tables

Multipart figures should be combined and labeled before final submission. Labels should appear centered below each subfigure in 8 point Times New Roman font in the format of (a) (b) (c).

## Referencing a Figure or Table Within Your Paper

When referencing your figures and tables within your paper, use the abbreviation "Fig." even at the beginning of a sentence. Do not abbreviate "Table." Tables should be numbered with Roman numerals.

## Submitting Your Graphics

Format your paper with the graphics included within the body of the text as you would expect to see the paper in print. Please do this at each stage of the review, from first submission to final files. For final files only, after the paper has been accepted for publication, figures should also be submitted individually in addition to the manuscript file using one of the approved file formats. Place a figure caption below each figure; place table titles above the tables. Do not include captions or borders in the uploaded figure files.

## File Naming

Figures (line artwork or images) should be named starting with the first 5 letters of the corresponding author's last name. The next characters in the filename should be the number that represents the figure's sequential location in the article. For example, in author "Anderson's" paper, the first three figures might be named ander1.tif, ander2.tif, and ander3.ps.

Tables should contain only the body of the table (not the caption) and should be named similarly to figures, except that '.t' is inserted in-between the author's name and the table number. For example, author Anderson's first three tables would be named ander.t1.tif, ander.t2.ps, ander.t3.eps.

Author photographs or biographies are not permitted in IEEE TMI papers.

## Checking Your Figures: The IEEE Graphics Analyzer

The IEEE Graphics Analyzer enables authors to pre-screen their graphics for compliance with IEEE Transactions and Journals standards before submission. The online tool, located at http://graphicsqc.ieee.org/, allows authors to upload their graphics in order to check that each file is the correct file format, resolution, size and colorspace; that no fonts are missing or corrupt; that figures are not compiled in layers or have transparency, and that they are named according to the IEEE Transactions and Journals naming convention. At the end of this automated process, authors are provided with a detailed report on each graphic within the web applet, as well as by email.

For more information on using the Graphics Analyzer or any other graphics related topic, contact the IEEE Graphics Help Desk by e-mail at email@example.com.

## Color Processing/Printing in IEEE Journals

All IEEE Transactions, Journals, and Letters allow an author to publish color figures on IEEE Xplore at no charge, and automatically convert them to grayscale for print versions. In most journals, figures and tables may alternatively be printed in color if an author chooses to do so. Please note that this service comes at an extra expense to the author. If you intend to have print color graphics, include a note with your final paper indicating which figures or tables you would like to be handled that way, and stating that you are willing to pay the additional fee.

# Some Common Mistakes

The word "data" is plural, not singular. The subscript for the permeability of vacuum $\mu _{0}$ is zero, not a lowercase letter "o." Use the word "micrometer" instead of "micron." A graph within a graph is an "inset," not an "insert." The word "alternatively" is preferred to the word "alternately" (unless you really mean something that alternates). Use the word "whereas" instead of "while" (unless you are referring to simultaneous events). Do not use the word "essentially" to mean "approximately" or "effectively." Do not use the word "issue" as a euphemism for "problem." When compositions are not specified, separate chemical symbols by en-dashes; for example, "NiMn" indicates the intermetallic compound Ni$_{0.5}$Mn$_{0.5}$ whereas "Ni–Mn" indicates an alloy of some composition Ni$_{x}$Mn$_{1-x}$.

Be aware of the different meanings of the homophones "affect" (usually a verb) and "effect" (usually a noun), "complement" and "compliment," "discreet" and "discrete," "principal" (e.g., "principal investigator") and "principle" (e.g., "principle of measurement"). Do not confuse "imply" and "infer."

Prefixes such as "non," "sub," "micro," "multi," and "ultra" are not independent words; they should be joined to the words they modify, usually without a hyphen. There is no period after the "et" in the Latin abbreviation "*et al.*" (it is also italicized). The abbreviation "i.e.," means "that is," and the abbreviation "e.g.," means "for example" (these abbreviations are not italicized).

A general IEEE styleguide is available at http://www.ieee.org/web/publications/authors/transjnl/index.ht ml.

# Conclusion

A conclusion section is not required. Although a conclusion may review the main points of the paper, do not replicate the abstract as the conclusion. A conclusion might elaborate on the importance of the work or suggest applications and extensions.

# Appendix and the Use of Supplemental Files

Appendices, if needed, appear before the acknowledgment. If an appendix is not critical to the main message of the manuscript and is included only for thoroughness or for reader reference, then consider submitting appendices as supplemental materials. Supplementary files are available to readers through IEEE *Xplore* at no additional cost to the authors but they do not appear in print versions. Supplementary files must be uploaded in ScholarOne as supporting documents, but for accepted papers they should be uploaded as Multimedia documents. Refer readers to the supplementary files where appropriate within the manuscript text using footnotes. [^5]

# Acknowledgment

The preferred spelling of the word "acknowledgment" in American English is without an "e" after the "g." Use the singular heading even if you have many acknowledgments. Avoid expressions such as "One of us (S.B.A.) would like to thank $\ldots$ ." Instead, write "F. A. Author thanks $\ldots$ ." In most cases, sponsor and financial support acknowledgments are placed in the unnumbered footnote on the first page, not here.

# References and Footnotes

## References

All listed references must be cited in text at least once. Use number citations that are placed in square brackets and inside the punctuation.

Multiple references are each numbered with separate brackets. When citing a section in a book, please give the relevant page numbers. In text, refer simply to the reference number. Do not use "Ref." or "reference" except at the beginning of a sentence: "Reference shows $\ldots$ .'

Reference numbers are set flush left and form a column of their own, hanging out beyond the body of the reference. The reference numbers are on the line, enclosed in square brackets. In all references, the given name of the author or editor is abbreviated to the initial only and precedes the last name. List the names of all authors if there are six or fewer co-authors, otherwise list the primary author's name followed by *at al.* Use commas around Jr., Sr., and III in names. Abbreviate conference titles. When citing IEEE transactions, provide the issue number, page range, volume number, year, and/or month if available. When referencing a patent, provide the day and month of issue or application. References may not include all information; please obtain and include relevant information. Do not combine references. There must be only one reference with each number. If there is a URL included with the print reference, it can be included at the end of the reference.

Other than books, capitalize only the first word in a paper title, except for proper nouns and element symbols. For papers published in translation journals, please give the English citation first, followed by the original foreign-language citation. See the end of this document for formats and examples of common references. For a complete discussion of references and their formats, see the IEEE style manual at https://journals.ieeeauthorcenter.ieee.org/your-role-in-article-p roduction/ieee-editorial-style-manual/.

## Footnotes

Number footnotes separately using superscripts.[^6] It is recommended that footnotes be avoided (except for the unnumbered footnote with the receipt date on the first page). Instead, try to integrate the footnote information into the text. Use letters for table footnotes (see Table ).

# References

-   *Basic format for books:*
    J. K. Author, "Title of chapter in the book," in *Title of His Published Book, x*th ed. City of Publisher, (only U.S. State), Country: Abbrev. of Publisher, year, ch. $x$, sec. $x$, pp. *xxx–xxx.*
    See .

-   *Basic format for periodicals:*
    J. K. Author, "Name of paper," *Abbrev. Title of Periodical*, vol. *x, no*. $x,$pp*. xxx–xxx,* Abbrev. Month, year, DOI. 10.1109.*XXX*.123456.
    See –.

-   *Basic format for reports:*
    J. K. Author, "Title of report," Abbrev. Name of Co., City of Co., Abbrev. State, Country, Rep. *xxx*, year.
    See .

-   *Basic format for handbooks:*
    *Name of Manual/Handbook, x* ed., Abbrev. Name of Co., City of Co., Abbrev. State, Country, year, pp. *xxx–xxx.*
    See .

-   *Basic format for books (when available online):*
    J. K. Author, "Title of chapter in the book," in *Title of Published Book*, $x$th ed. City of Publisher, State, Country: Abbrev. of Publisher, year, ch. $x$, sec. $x$, pp. *xxx–xxx*. \[Online\]. Available: http://www.web.com
    See –.

-   *Basic format for journals (when available online):*
    J. K. Author, "Name of paper," *Abbrev. Title of Periodical*, vol. $x$, no. $x$, pp. *xxx–xxx*, Abbrev. Month, year. Accessed on: Month, Day, year, DOI: 10.1109.*XXX*.123456, \[Online\].
    See –.

-   *Basic format for papers presented at conferences (when available online):*
    J.K. Author. (year, month). Title. presented at abbrev. conference title. \[Type of Medium\]. Available: site/path/file
    See .

-   *Basic format for reports and handbooks (when available online):*
    J. K. Author. "Title of report," Company. City, State, Country. Rep. no., (optional: vol./issue), Date. \[Online\] Available: site/path/file
    See .

-   *Basic format for computer programs and electronic documents (when available online):*
    Legislative body. Number of Congress, Session. (year, month day). *Number of bill or resolution*, *Title*. \[Type of medium\]. Available: site/path/file
    ***NOTE:* ISO recommends that capitalization follow the accepted practice for the language or script in which the information is given.**
    See .

-   *Basic format for patents (when available online):*
    Name of the invention, by inventor's name. (year, month day). Patent Number \[Type of medium\]. Available: site/path/file
    See .

-   *Basic formatfor conference proceedings (published):*
    J. K. Author, "Title of paper," in *Abbreviated Name of Conf.*, City of Conf., Abbrev. State (if given), Country, year, pp. *xxxxxx.*
    See .

-   *Example for papers presented at conferences (unpublished):*
    See .

-   *Basic format for patents*$:$
    J. K. Author, "Title of patent," U.S. Patent *x xxx xxx*, Abbrev. Month, day, year.
    See .

-   *Basic format for theses (M.S.) and dissertations (Ph.D.):*

    1.  J. K. Author, "Title of thesis," M.S. thesis, Abbrev. Dept., Abbrev. Univ., City of Univ., Abbrev. State, year.

    2.  J. K. Author, "Title of dissertation," Ph.D. dissertation, Abbrev. Dept., Abbrev. Univ., City of Univ., Abbrev. State, year.

    See .

-   *Basic format for the most common types of unpublished references:*

    1.  J. K. Author, private communication, Abbrev. Month, year.

    2.  J. K. Author, "Title of paper," unpublished.

    3.  J. K. Author, "Title of paper," to be published.

    See –.

-   *Basic formats for standards:*

    1.  *Title of Standard*, Standard number, date.

    2.  *Title of Standard*, Standard number, Corporate author, location, date.

    See .

-   *Article number in reference examples:*
    See .

-   *Example when using et al.:*
    See .

[^1]: This paragraph of the first footnote will contain the date on which you submitted your paper for review. It will also contain support information, including sponsor and financial support acknowledgment. For example, "This work was supported in part by the U.S. Department of Commerce under Grant BS123456."

[^2]: The next few paragraphs should contain the authors' current affiliations, including current address and e-mail. For example, F. A. Author is with the National Institute of Standards and Technology, Boulder, CO 80305 USA (e-mail:firstname.lastname@example.com).

[^3]: S. B. Author, Jr., was with Rice University, Houston, TX 77005 USA. He is now with the Department of Physics, Colorado State University, Fort Collins, CO 80523 USA (e-mail: firstname.lastname@example.com).

[^4]: T. C. Author is with the Electrical Engineering Department, University of Colorado, Boulder, CO 80309 USA, on leave from the National Research Institute for Metals, Tsukuba, Japan (e-mail: email@example.com).

[^5]: Supplementary materials are available in the supporting documents/multimedia tab. Further instructions on footnote usage are in the Footnotes section on the next page.

[^6]: Place the actual footnote at the bottom of the column in which it is cited; do not put footnotes in the reference list (endnotes).