Skip to content

UI Design

Jump to bottom Edit New page
Ilya Sher edited this page Mar 3, 2024 · 42 revisions

UI Software Design

This is more of a technical / brain-dump document. For more coherent description of identified problems with UIs of other shells and the vision for the solution in NGS see UI in NGS blog post.

Shells' UI Problem

Current shells as well as proposed alternatives treat UI as if nothing happened since the 70-s. The UI is stuck in telegraph-like communication paradigm: send text - receive text. It's not a coincidence; it's a historical development. The time to rethink the UI was at 1976, but I guess now is the second-best time. Watch the 15 minutes for more details about what's wrong and how NGS is solving that.

Unix shell - We can do better now

Old Demo

See screencast of small-poc is on youtube: http://www.youtube.com/watch?v=T5Bpu4thVNo (very dated).

Requirements

Semantic Widgets

All visible objects on the screen (widgets) should carry semantic information with them whenever possible. Semantic information will allow various interactions with the widgets, such as appropriate context menu.

Widgets should have unambigous references to the underlying objects they represent. For example, for EC2 instance that would be AWS account id + region + instance id. ARN format would probably be good for interoperability.

  • Sharing a widget with another person should be possible (more about this in Timeline Design - Sharing)
  • All objects/widgets must have unique ID to assist cross-session correlation.
  • The objects will have misc. semantic information to allow plugins to provide functionality on different semantic layers:
    • Attributes
    • Type of the object (ex: AWS::CodePipeline::Pipeline, exact format TBD)
    • Field-of-object: id, type, field, relation_type (ex: AWS::EC2::Instance, field VpcId, rel contained_in)
    • Command that was used to obtain (ref and/or argv, ex: [aws,ec2,describe-instances] or something more semantically meaningful)
    • Context
    • Chain
    • ...
  • Objects will have defined and exposed schema
  • Plugins should be able to provide additional fields for each kind of object.
    • Ex: CodePipeline was created by you (the user).
    • Ex: Commit was authored by you
  • Make sure there is full separation between the data and presentation.

Chains

  • A chain is a sequence of operations to accomplish a task.
  • A chain carries semantic meaning (each step carries semantic meaning too).
  • A chain is constructed by recording user interaction with the widgets.
  • Code can (and should) be generated from a chain.
    • The UI should try to detect high level patterns
      • Example: detect CDK L2 Constructs
  • There should be an attempt to auto-name chains, maybe suggest a list of alternatives

Sample chain:

  • List AWS profiles
  • Select profile experiments (Context)
  • List AWS (enabled) regions of the profile
  • Select eu-central-1 region (Context)
  • List CloudfFormation stacks in the region
  • Select CloudfFormation stack named ilya-dev
  • List resources in the stack
  • Select resource with logical name AuthLambda
  • Go to the selected Lambda logs
  • Tail the logs (all log groups aws logs tail $LOG_GROUP --follow)

Sample chain steps:

  • Select the only AWS VPC where IsDefault is true
  • Select the only AWS EC2 instance tagged Name=ilya-01
  • Select the most recently launched AWS EC2 instance
  • Follow all security groups of an AWS EC2 instance
  • Show CloudFormation responsible for a resource following the aws:cloudformation:stack-name tag

Session

Support of the "session" concept - a work on particular task by the operator.

  • Ability to switch to a session at any time, even after exiting and starting the shell.
  • Separate per-session history, so it could be shared, removed, etc. See also Timeline Design.
  • Ability to re-play session, step by step, optionally modifying commands.
  • Continuing session after exiting and restarting the shell would be possible

Reproducibility

All operations made via the UI, including mouse operations in GUI must have and display textual representation, allowing: copy / paste / save to a file / send to friend / log in history

  • All widgets displayed in the UI must be results of commands with textual representation
    • Example: if a user sorts a table using the UI, a command must be generated and run to display the sorted results
    • All widgets must carry with them the commands that generated them. More specifically a history entry which includes whatever is necessary to reproduce the command (working directory, environment variables, etc). Maybe standard input... (and output for history purposes).
    • In addition to textual representation, it's probably beneficial to keep semantic structured data about the selection/action event

UI Pluggability

Ability to support several i/o modules such as terminal (CLI) and web.

API

Provide API to NGS scripts (and other scripts via CLI) to create/update widgets.

  • Example: API/protocol to report the progress
    • Current task (Copying mydata.txt to /tmp/)
    • Overall progress (70% or File 7 out of 10)
    • ETA maybe

View / Interaction - General

TODO: Review this section.

  • "Training wheels" - configurable amount of help for beginners (similar to Lynx browser). Examples:
    • Shortcuts can be shown
  • Interactivity
    • Keep in mind that many of the widgets will be interactive, not just a static text dump
    • Interactive data types
      • Protocol for specifying operations on the interactive data types
      • Should gracefully degrade when go to file or piped to another program instead of interactive UI
    • Allow navigation on the screen, choosing operations from a menu for the objects
    • Allow selection of multiple objects and performing an operation on all of them
      • The generated code will have a loop or use a facility for parallel execution (contextually derived or user provided which one)
    • History of operations in context menu - all actions must have textual representation. This enables history.
    • Interoperability with non-NGS scripts. External programs should be able:
      • Detect running under NGS
      • Return interactive objects
  • Display structured results as such (JSON, Yaml, ...)
    • Most of the data dealt with is tables. List of files, list of instances in a cloud, list of load balancers. Amazing that none of current UNIX shell tools (I heard of) don't treat the data as such. The closest you get is set of records in awk. Well, if the fields in records are the same it's actually a table. $1 in awk could be id or name, referencing the data by column name and not by field number. Yes, you have jq and it's close but it still works (in best case) with list of records with same fields. PowerShell has something in that direction (Get-Process | Where-Object {$_.handles -gt 200}) in combination with Format-Table.
    • (Maybe) Allow editing it and saving to file.
    • (Maybe) Allow write jq filters in (G)UI by selecting the elements
  • Easy navigation to related objects (EC2 instance <-> Containing VPC; EC2 instance <-> attached EBS; EC2 instance <-> attached subnet)
  • Smart output grouping for large outputs. For example, if there are more than X EC2 instances are listed, they will not be shown by default but rather a summary. The summary will allow further navigation using additional filtering on the original query. Example output for large number of EC2 instances: ""You have listed 1000 instances. Suggested filters: (A) 900 t1.small , 100 c3.medium (B) 800 env=prod , 200 env=Dev, ...". Low cardinality filters first?
  • Tables should support
    • different types of resources in one table
    • different parts of table generated by different commands?
    • grouping (including by things like cost)
  • Search box that also decodes (example: JWT)
  • Convert output. Example: use pandoc to convert man to html.

Unsolicited Information

Semantic breakthrough: showing important information automatically. In classical shells, there is no way for "unsolicited" information to appear. Imagine process running in a background with output mixed with everything else. Because shells don't separate the outputs.

Programs that show unsolicited information should be called "Timeline Contributors".

  • There should be a list of Timeline Contributors with on/off switches and their settings.
  • CI/CD run, errors, failed health checks, etc should be shown automatically. Filter by current context or defined "interests" (possibly list of contexts or something like "all AWS profiles").
  • Each such notification should have information about which Contributor it comes from and why it is shown. Ex:
    • Showing CodePipeline execution because you are author of the triggering commit
    • Showing CodePipeline execution because you created the CodePipeline
    • Showing CodePipeline execution because it failed
    • Showing CodePipeline because it's waiting for manual approval
  • When some kind of notification is turned off, there should be a convenient way to turn back on. Probably on the plugin page.
  • Clearly mark indirectly started processes (such as CodePipeline execution triggered by a commit)
  • For critical (time sensitive) information, show last update time.

Context Area

  • There should be one area which would hold widgets for current contexts
  • Examples of contexts
    • Current directory
    • Current AWS account/profile (open issue - support for multiple)
    • AWS STS assume role (minds the expiration)
    • AWS tags - to narrow down any listing of AWS resources to resources with specific tags
    • Current Git repository?
    • Resource(s) in a cloud?
    • Jetbrains project context (so menu -> open in Jetbrains)?
  • Context widget should have a history (LRU) so that switching to previous context would be easy. This is per context type history.
  • Bookmarks?
    • Including to point in the Timeline
  • Copy/paste buffers?
  • Temporary and/or open files?
  • Contexts should have stable/predictable order on the screen

See Context Design

Visualization Area

  • Visualization area shows diagrams (SVG? Text?) for easy navigation to related objects.
  • Visualization area can contain different views. Examples:
    • Network (EC2 instance -> subnets, VPC)
    • Security (EC2 instance -> security groups, instance profile)
    • Relevant CloudWatch graphs

Commands Area and Processes handling

TODO: Review this section.

  • Commands prompt and input area

    • The prompt should be interactive (for example, click on git branch name in prompt -> context menu -> "change branch" -> branches menu -> pick one)
    • The prompt should be dynamically-updatable (for example would poll something and display result)
    • Notification widgets, so that different plugins/components have a place to report the status. Notification widgets must be interactive. Example of notifications widgets which could be implemented:
      • Build (in CodeBuild or Travis CI) succeeded/failed.
      • CloudWatch alarms
      • CloudTrail events notifications
      • Remote git branch got new commits
      • SNS subscription
      • Polling (SSH) a server till some condition is true, such as provision finished
      • Polling a server till SSH connection is possible
    • Commands scroll up, new commands are added at the bottom. When a command that haven't completed yet, reaches top of the screen, it can be converted to a mini-area at the top (right?) of the screen, representing the command and current progress (and exit status later).
    • Dynamic update command output, think ps / top when their output is in the stream
    • "pin" feature so user defined command sticks to the screen and being re-run and the output updated, essentially making it a widget

  • Each process will have its own area on the screen

    • Process view shows current subprocess in a tree like fashion
    • Expanded view (on the right presumably) shows past and future subprocesses.

  • Not to block, allow typing next commands even if previous command is still running

    • By default, consider commands related and only queue next commands but do not execute
    • Upon successful completion of a command, execution of the next one starts
    • If an error occurs, pending commands are blocked, until the user resolves the situation and reruns them
    • There should be an option not to queue the newly entered command but to start the execution immediately

  • Output

    • Truncated stdout/stderr: Commands' outputs displayed below the commands up to max N lines then scroll the output in a small window below the command. When there is more output than a human can process - don't display it and suggest saving it to a file (maybe).
    • Option to view full stdout/stderr in a pager
    • Ability to mark "seen until here" in output, similar to pressing enter several times in a terminal while a command is running.
    • Preferred fields order in the output (defaults to "smart" order)
    • Output should contain as much meta-information as possible
      • Ex: from which low level command each piece of information came from.
      • Ex: JSON path / jq expression / NGS expression to get to the element you are hovering over

  • Provide good feedback. In GUI for example, this can be green / red icon near a completed command to show exit status. Tweaking prompt to include such info or typing echo $? all the time is not what I dream about.

    • External, unaware programs:
      • Easy way to write hooks that will provide additional information about running process, such as progress.
      • For external programs that will be unaware of any progress reporting protocol (currently all existing programs), use heuristics such as look at open files + position in files to guess the progress with high probability.

  • Commands history: among duplicate commands all but last should be grayed out, so that non-grayed out commands are unique.

  • When hover over an object, highlight the same object everywhere it appears on the screen.

  • [later] Confirmation mode. One user in collaboration mode gives the command to execute, another user must approve the command for execution.

  • [later] Underline red/green for existing/non-existing files? Fish shell does it for the commands.

  • Maybe: allow automatic chaining of monitoring commands. Example: after call to async command, automatically execute polling command until the status change from semantic "in progress" to something else (success/failure).

Completion

TODO: Review this section.

  • Smart completion, context sensitive

    • Command switches and values
    • Complete objects (file names, urls, etc) depending on context. Think wget .../x.tgz, tar [COMPLETION_KEY] [Maybe some choice keys] -> xzf x.tgz
    • Maybe API to insert objects to completion history
    • Auto-detect completion history objects for existing commands (by wrappers and/or hooks probably)

  • "Mentioned" completion

    • Complete objects from output of previous commands. Example: apt-cache search ... , apt-get install ... Isn't this copy+paste annoying? It's already on the screen, it's a package name, and still the system can't complete...

Open issues

  • How to deal with a command that requires regular stdin/stdout interaction.
  • Maybe: API/protocol for interaction
    • Maybe using file descriptors (must be stdin/stdout in case of remote shell?)
  • How to treat timestamps in outputs? Jump to that time in the Timeline? Is that helpful?

Ideas

  • In a table, each cell can have the following uses (others too?) :
    • Serve as filter for the list of the objects displayed in the table
    • Serve as navigation to another type of object
  • Table columns can have suggestions for filtering. Ex: EC2 - latest launched, CodePipeline - failed status, CodePipeline - in progress.
  • Each piece of information (widget and others) must have priority associated with it. This will allow displaying above a threshold and "sticky" display for very important information.
  • UI representation of a job. A map with all involved processes, their open files, sockets, pipes, resource usage (CPU, disk, network), process running time, accumulative CPU time, ...
  • Groups of resources
    • Groups of resources defined by predicate
    • Select groups of resources. Examples
      • All resources of a CloudFormation stack
      • Equivalent resources in another stack/az/region/account
      • Having same type of relation to another resource: same IAM role, on same subnet, in same VPC, etc
    • Compare groups of resources
  • "limits" view
  • "compliance" view
  • CloudTrail as a source for Timeline
  • Popup preview of objects on hover, like Wikipedia does
  • Incorporate manual tweaks to generated commands
  • Clipboard with named items
  • Last seen marker like in a chat
  • When starting a session, show "while you were missing" list of interesting events that occurred after last session and before current session.
  • Last used objects
  • Looking at IaC declaration allow navigation to instances (deployments) of the declared resource
  • A way to express intent, which might mean recommendation of few commands to achieve the goal
  • Detect from beginning of a chain what you are trying to accomplish. Suggest next steps of premade chain.
  • Understanding link between output of a command and all possible commands that have as possible input any of the output types of the first command.
  • Show deeper information. Example: in list of AWS profiles, show which ones have problems such as failed build

Terminal integration

TODO: Review this section.

  • Bidirectional metadata communications channel with the shell (for iTerm2 integration and possibly others). Most of the points below are rephrased email from George Nachman, creator of iTerm2.
    • Check Protocol Buffers
    • New DCS code to communicate capability of the shell and then switch protocols
      • Need to make sure that all output is captured and goes through the new protocol
    • Features
      • Possible marking of output metadata such as
        • Belonging to a particular process
        • Output file descriptor (stdout/stderr) on which the data came out of the process
        • Mark objects such as files so that terminal could provide context menu with file operations for example.
        • In general, pass all possible metadata about the objects on the screen
      • Tell terminal about running jobs so it could present a task manager (plus maybe UI to select active task on the terminal side)
      • Typing a command while a command is running in cooked mode could have a nice UI rather than mixing command output with echoed keystrokes
      • File transfer between terminal and the shell
      • Status bar
      • Semantic history (need to check how relates to History Design)
      • Interactive commands on the objects on the screen
        • Shell sends context menu for each object (or semantic object type) to the terminal
        • Terminal sends the shell menu interaction result

Confined output of a program

External programs' outputs will be confined. The main readme contains many ideas about this (which should be move from readme to this page).

Related Projects

Design (WIP)

TODO: Overall description

Components (WIP)

TODO: List of components. For each: description of the role, API, etc.

  • Session manager - anything pertaining to history and managing the whole list of widgets
  • Layout manager - organizes widgets on a canvas

Types and methods (WIP)

  • UI::SessionManager

    • Fields
      • TBD
    • Methods
      • push(SessionManager, Widget) ?

  • Widget type. (CommandsPipelineWidget, CommandWidget, etc subtypes)

    • Fields
      • id - globally unique widget id, probably UUIDv4, maybe URI (ngs:widget:SESSION-ID:WIDGET-ID)
      • uoid - underlying object id
      • meta - a Namespace
        • created - Time type object
        • updated - Time type object
      • data - a Namespace of fields, subtype-specific
      • children - sub-widgets. Note that displaying order of children is independently configurable, somewhat like CSS.
    • Methods
      • render(Widget, ...). Parameters TBD. Candidates: constraints (for dimensions etc), target (for ), context. Returns target-specific data structure. For a terminal that would be the text lines for example. The output might be even more specific depedending on for example whether ther terminal supports UTF-8.

Resources

Terminal Libraries / Applications

Terminals

These terminals have good working image support with either sixel or iTerm2 image formats. (Kitty images are only supported by kitty and wezterm, and as of February 2022 do not always behave the same way.)

Widget Examples

NGS official website is at https://ngs-lang.org/

Add a custom sidebar
Clone this wiki locally