Goal Programming

Case Study: Project Management with Shortcut (WIP)

Shortcut is a project management SaaS product with feature-set tailored to medium-sized teams. In this case study, we'll write code to interoperate with Shortcut's API to create some sample data, retrieve it via Shortcut's REST API, and perform project management data analysis.

System Setup

First, ⎘ install Goal locally.

Second, install the curl command for your system (we'll shell out to that to make HTTP calls).

Next, sign up for Shortcut (it's free for what we'll use). Once you're logged into your Shortcut workspace, go to Settings > API Tokens and make an API token that you save to a SHORTCUT_API_TOKEN variable in your shell environment.

Sample Data

Download the sample JSON that contains a set of completed stories from an epic (a story is a unit of work, an epic is a collection of related stories, generally a single overall deliverable with a start and end date).

Note: Since you already have the data set in your hands, if you're more interested in the data analysis portion of this case study you can skip to Data Analysis.

Here's an example of one story from the data set:

 {
  "story_links": [
   {
    "entity_type": "story-link",
    "object_id": 277551,
    "verb": "blocks",
    "type": "object",
    "updated_at": "2024-11-04T15:45:19Z",
    "id": 500176162,
    "subject_id": 277549,
    "subject_workflow_state_id": 500180258,
    "created_at": "2024-11-04T15:45:19Z"
   }
  ],
  "task_ids": [],
  "story_type": "chore",
  "workflow_id": 1488,
  "completed_at_override": null,
  "started_at": null,
  "completed_at": null,
  "requested_by_id": "5c473fd4-c531-45c3-8203-c1798e99b030",
  "sub_task_story_ids": [],
  "started_at_override": null,
  "group_id": null,
  "workflow_state_id": 500144468,
  "owner_ids": [],
  "id": 277551,
  "parent_story_id": null,
  "estimate": 2,
  "deadline": null,
  "created_at": "2024-11-04T15:44:45Z",
  "name": "Sit amet consectetur"
 }

The code that follows will allow you to populate your own Shortcut workspace with this sample data. You can then use Shortcut's web app to add or change story estimates, organize stories with labels, or put the stories into different workflow states to model different stages of completion.

Shortcut's API allows you to override the "started at" and "completed at" values for a story, specifically for use-cases like importing project management data from other platforms, so you can model many scenarios without having to wait for actual time to pass.

Here's the complete program, we'll pull it apart piece by piece below:

url:"https://api.app.shortcut.com/api/v3"; token: 'env"SHORTCUT_API_TOKEN"; headers:qq#-H "Shortcut-Token: $token" -H "Accept: application/json" -H "Content-Type: application/json; charset=utf-8"#
tmpfile:"/tmp/tmp.json"
curl:{[url]qq#curl -sS $headers "$url"#}
curlp:{[url;body]jb:""json body;tmpfile print jb;qq#curl -L -XPOST $headers -d @$tmpfile "$url"#}
ss:json read "case-study-epic-stories.json"
members:json shell curl"$url/members"
owners:?,/(..x["owner_ids"])'ss
requesters:?,/(..x["requested_by_id"])'ss
memberks:?owners,requesters
membervs:(#memberks)#(..id)'members@&"full"=(..state)'members
membermap:memberks!membervs / data set members to new workspace members
workflowmap:(@[;"id"]'wfs)!wfs:json shell curl"$url/workflows"
wfid:*&"TestDaniel"=(..name)'workflowmap
wfstid:*(..id)'states@&"Completed"=(..name)'states:workflowmap[wfid;"states"]
lbl:{ts:time"unixmilli";..[name:"case-study-goal-$ts";color:"#00b478";description:"Label for stories imported as part of 'Case Study: Project Management with Shortcut' on GoalProgramming.info"]};lbl""
res:shell curlp["$url/labels";lbl""]
remotelbl:json res; lblname:remotelbl"name"
newstory:{[lblname;oldstory](name;st;ca;sa;r;os;estimate;deadline):oldstory[!"name story_type completed_at started_at requested_by_id owner_ids estimate deadline"];..[name;"story_type":st;"workflow_state_id":wfstid;"completed_at_override":ca;"started_at_override":sa;"requested_by_id":membermap[r];"owner_ids":membermap[os];estimate;deadline;labels:,..["name":lblname]]};newstory[lblname;(*ss)]
bulkres:shell curlp["$url/stories/bulk";..[stories:newstory[lblname]'ss]]

The first four lines define an HTTP client for the Shortcut REST API, using your system's installation of curl.

Let's review the first line in detail:

url:"https://api.app.shortcut.com/api/v3"; token: 'env"SHORTCUT_API_TOKEN"; headers:qq#-H "Shortcut-Token: $token" -H "Accept: application/json" -H "Content-Type: application/json; charset=utf-8"#

The first line defines url, token, and headers variables. The url is the base URL that will be used for all Shortcut API calls. The API token is expected to be found in the environment as a variable called SHORTCUT_API_TOKEN, so that env can find it by that name. The env call is prefixed with a single apostrophe ' so that this program returns an error if that lookup fails. Finally, the HTTP headers necessary to authenticate with Shortcut's API and have our payloads recognized as JSON are defined. I use the qq#...# syntax for the string to be able to use a delimiter other than quotation marks, while also being able to interpolate variables (use rq for completely raw strings with custom delimiters).

The second line defines a tmpfile variable, which we'll use to store JSON that we want to POST to Shortcut API endpoints. I've put it on its own line so the user notices it, since the program makes no effort to clean up this file.

The third and fourth lines define functions curl and curlp for crafting GET and POST requests respectively:

curl:{[url]qq#curl -sS $headers "$url"#}
curlp:{[url;body]jb:""json body;tmpfile print jb;qq#curl -L -XPOST $headers -d @$tmpfile "$url"#}

The curl function takes a url argument and puts it into a string for a curl command with flags to make curl's output quiet except for the response body.

The curlp function takes a url and a Goal data structure as body, writes the Goal data structure to the tmpfile, and then returns a string representing a curl command to make the POST call with the contents of tmpfile as the request body.

Note: Both of these functions are pure, returning a string representing the curl command. To run the curl command, we'll use Goal's shell verb, which we'll see in action a few times in this program.

Now with our primary utility functions in place, we can get to work on the data.

The next line creates a variable called ss (mnemonic for "stories") by parsing a local copy of the JSON in a file called case-study-epic-stories.json. If we wanted this program to be more robust to errors, we could add ' before both the read and json calls, so that the program would exit early on an error. Since I produced this JSON myself and have it on my file system, I didn't bother to do this.

The story data has requested_by_id and owner_ids fields, which contain UUIDs for user identifiers from the original Shortcut workspace. These UUIDs won't be valid in our target Shortcut workspace, so we'll need to create a mapping from the old IDs to the new ones. We'll start by getting all of the users (Shortcut calls these "members") who are a part of the target workspace:

members:json shell curl"$url/members"

For the purposes of this analysis, it doesn't matter which members are mapped to which, so we'll gather all of the unique IDs across requesters and owners (N total UUIDs):

owners:?,/(..x["owner_ids"])'ss
requesters:?,/(..x["requested_by_id"])'ss
memberks:?owners,requesters

The ..x["owner_ids"] syntax creates a function that looks up "owner_ids" in the dictionary passed as an argument. We run that function over each ' of the stories in ss, which returns a list of owner ID lists (i.e., a list of lists of UUIDs). Since we're just interested in unique UUIDs, we can concatenate all of those sub-lists together by using join , with fold / to join all of the lists together into one flat list of UUID strings. The ? function returns the unique set of UUIDs from that overall list.

The same thing is down for requesters. I bound these separately because I was interested in how many owners vs. requesters there were. The memberks variable (mnemonic for "member keys") uses the same computation, returning the unique items from joining the owners and requesters lists.

Then we create a mapping from them to the first N members in the target workspace:

membervs:(#memberks)#(..id)'members@&"full"=(..state)'members

membermap:memberks!membervs / data set members to new workspace members

The following three lines use mostly the same constructs. For some reason I used @[;"id"] instead of (..id) (equivalent in this case), but otherwise you should be able to apply what you learned above to read these.

After that, I define a lbl function to produce a dictionary in the shape that Shortcut's API expects if you want to create a new label in your Shortcut workspace. You can tag stories (and epics) with labels in Shortcut to organize your work, and we'll add one so that if we make a mistake or simply want to clean up after ourselves, we can look up the stories we've created by label to do so.

lbl:{ts:time"unixmilli";..[name:"case-study-goal-$ts";color:"#00b478";description:"Label for stories imported as part of 'Case Study: Project Management with Shortcut' on GoalProgramming.info"]};lbl""

The lbl function is define on one line. It's defined as a function so that each time it's invoke it produces a current timestamp value (which is interpolated into the name of the label). A call to time"unixmilli" gives us milliseconds since the Unix epoch, saved as the ts variable within the function definition. The ..[name:"...";color:"...";description:"..."] syntax creates a dictionary (see the parallel with how ..name would create a dictionary lookup function, but ..[name:"..."] creates a dictionary itself?). Finally, on the same line, I use a semicolon and add a call to this function with lbl"", so that when I (re)evaluate this line of code, I can see what the function's return value looks like. Goal doesn't support true niladic (zero-argument) functions, so "" is a throw-away argument in this case.

💡Even though you'll learn that the colon : verb can be used to print out the value of a variable at the REPL (like :a:42 which prints 42, instead of just a:42 which prints nothine), when you do that in a script and run the script, that actually causes your script to exit early at that point, which confused me several times before I realized what was happening. For that reason, I use the above code pattern of including an example invocation on the same (or last) line when defining a function during development.

The definition of newstory includes the syntax for binding several variables at once using an array as the right-hand of :, and it also shows the ergonomic shortcut of ..[name;estimate;deadline] syntax whereby if the name of a variable and a key match, you can simply include the variable without having to do ..[name:name;estimate:estimate;deadline:deadline]. Finally, it uses , monadically to create a list from a single dictionary (where we only used it dyadically above). Otherwise, you should be able to read the rest of the program without a step-by-step walk-through.

Data Analysis (TODO)