A Custom Browser Start Page

The reality of my job as an engineering manager involve closely monitoring everything happening on Github,
For this purpose I watch all my company repos and I take the habit, every day to go through my notifications inbox and clear everything.
This process is very tedious as I always have at least 24-50 items every day waiting to be reviewed.

Here is my workflow:

1. I join this interface
   
2. login to my github org SSO
   
3. Reach the Next button until there is no more to sort the list by chronological order
   
4. Go through each Notification, treat it
   
5. Mark it as done
   

This feature is nice but I wish I could solve a couple things:

1. Each page shows 50 items max, I would like to see everything in a single page
   
2. I want to sort everything by chronological order without having to click a button multiple times
   
3. Marking something as done is OK but perhaps I just want to dismiss it as soon as I visit the notification
   

Github exposes most of its functionalities through its API so maybe I can come up with something tailored for my needs, let's try something!

Head over to Github.com then profile > Settings and select Personal access tokens then generate a new access token.

Fill in a note and select the required scopes, for my purpose I only need notifications for now.
Finally select Generate Token optionally bless the new token with your organization SSO auth if needed.
Carefully save the password to a secure place (encrypted).

I will use Clojure because I don't know exactly where I'm heading with this project and I like the idea of a living material that I can manipulate and experiment at will through the repl.

For this project I would like to start with something minimal, I will spin a classic web server and generate the HTML (no SPA).
I may reconsider this choice later but I think a browser start page has to render fast and not involve too much client side scripting.

I will include very few dependencies, I basically need only 3 things:

1. A web server and a routing library (ring)
   
2. Something to make HTTP requests (clj-http)
   
3. Something to template the data and render HTML (hiccup)
   

  (defproject oracle "0.1.0-SNAPSHOT"
    :description "A simple Browser Home page in Clojure"
    :url "http://github.com/gbuisson/oracle"
    :license {:name "EPL-2.0 OR GPL-2.0-or-later WITH Classpath-exception-2.0"
              :url "https://www.eclipse.org/legal/epl-2.0/"}
    :dependencies [[org.clojure/clojure "1.10.1"]
                   [clj-http "3.10.1"]
                   [cheshire "5.10.2"]
                   [hiccup "1.0.5"]
                   [ring "1.9.5"]
                   [clojure.java-time "0.3.3"]
                   [org.ocpsoft.prettytime/prettytime "5.0.2.Final"]]
    :repl-options {:init-ns oracle.core
                   :host "0.0.0.0"})

Querying the Github API is pretty simple, just send json requests wrapping your auth as basic:

(defn github-request [{:keys [handle token]} request]
  (client/request (into request
                        {:as :json
                         :basic-auth [handle token]})))

I don't like the fact that I send my static token with each HTTP request, I would prefer trading my token once every session and get a short lived access token as it makes it harder for an attacker to retrieve it if the connection is compromised.
From here the notification API endpoint is straightforward, I'll just create a small function with an options map and a middle function to separate the data extraction logic in case it grows in complexity:

  (defn get-notifications-url [{:keys [github-api-url]}]
    (str github-api-url "/notifications"))

  (defn get-notifications [config]
    (github-request config
                    {:method :get
                     :url (get-notifications-url config)
                     :as :json}))

  (defn fetch-unread-notifications [config]
    (:body (get-notifications config)))

Every Clojure web server starter just does the same thing, defonce the server so it doesn't conflict when you reload the repl,
Create a bare minimal handler with a simple function that always return the same thing for each request:

(defn make-handler [config]
  (fn [request]
    (let [unread-notifications (fetch-unread-notifications config)]
      {:status 200
       :headers {"Content-Type" "text/html"}
       :body (html (render-page unread-notifications))})))

(def handler
  (make-handler config))

(defonce server
  (jetty/run-jetty
   (wrap-reload #'handler)
   {:port 3000
    :host "0.0.0.0"
    :join? false}))

Rendering the data with hiccup is fabulous for the simple reason that you don't manipulate string but simple data structures,
you can easily assemble multiple small blocks using small functions.

A couple things:

1. I will simply display 2 sections, one for the PRs and another for the Issues
   
2. I will display each item as a link displaying the time and the item type
   

  (defn render-notifications [user unread-notifications]
    (let [groups (group-by #(-> % :subject :type keyword) unread-notifications)
          issues (get groups :Issue)
          prs (get groups :PullRequest)]
      [:div {:style "width:50%;float:left;"}
       [:div
        [:h2 "Github PRs "
         [:span (format "(%s)" (count prs))]]
        [:ul (for [{:keys [subject repository]
                    :as notification}
                   prs]
               [:li
                (:updated_at notification) " - "
                (:type subject) " - "
                [:a {:title (:name repository)
                     :target "_blank"
                     :href (notification-link user notification)}
                 (-> notification :subject :title)]])]]

       [:div
        [:h2 "Github Issues "
         [:span (format "(%s)" (count issues))]]
        [:ul (for [{:keys [subject]
                    :as notification}
                   issues]
               [:li
                (:updated_at notification) " - "
                (:type subject) " - "
                [:a {:target "_blank"
                     :href (notification-link user notification)}
                 (-> notification :subject :title)]])]]]))

      (defn render-page [user unread-notifications links]
            [:div (render-notifications user unread-notifications)])

The Github Notifications management features are unfortunately not exposed publicly on their API.
When I browse the notification url I only see the item and nothing to dismiss it.
If I want to display the Done button I need to reverse engineer how it's done.

Comparing the urls provided by the API responses and the ones I see browsing the interface I see a specific query argument notification_referrer_id.
A quick search on Google and similar Github projects indicate that it's a secret magic string you need to compose this way:

1. gather the user id and the notification id
   
2. Prefix it with a magic string 018:NotificationThread
   
3. Encode it as base64
   

I will also add some quick fixes I found along the way to generate correct links to the Github interface.

  (defn notification-referrer-id [notification-id user-id]
    (let [id-string (str "018:NotificationThread" notification-id ":" user-id)]
      (.encodeToString (Base64/getEncoder) (.getBytes id-string))))

  (defn notification-link [user {:keys [id subject]
                                 :as notification}]
    (let [transformed-url
          (-> (:url subject)
              (string/replace
               "api.github.com/repos" "github.com")
              (string/replace "pulls" "pull"))]
      (str transformed-url "?notification_referrer_id="
           (notification-referrer-id id (:id user)))))

The first time I generated my page I only witnessed 50 notifications which sound both fishy and small.
Obviously Github doesn't send all the results on one HTTP Response but includes a handle Link HTTP header which countains a next reference if one need to paginate.

Now is the time to increase the complexity of my fetch-unread-notifications function.

I need to make it scroll every page of the paginated response which you can do very easily in clojure using loop.
I should add a config argument to specify a maximum number of pages to scroll as a safeguard.

basically:

1. start a loop with an accumulator for the notifications (empty by default) and an increment
   
2. Execute the query and concatenate the results
   
3. If there is a next page continue up to 50 pages by default
   
4. otherwise return only the accumulated results
   

like this:

(defn fetch-unread-notifications
  [{:keys [max-notification-pages]
    :or {max-notification-pages 50}
    :as  config}]
  (loop [notifications []
         page 1]
    (let [{:keys [body headers]}
          (get-notifications config {:query-params {:per_page 50
                                                    :page page}})]

      (if (and  (clojure.string/includes? (get headers "Link") "next")
                (<= page max-notification-pages))
        (recur (concat body notifications) (inc page))
        (concat body notifications)))))

This gets me 249 notifications which does look like a more realistic figure :-)

Now I can launch a repl with this project, it serves me a page on http://localhost:3000, I immediately see a list with all my notifications, I can set that as my start page so every time I open a browser tab I notice new stuff.
I prefer this approach over a popup so I don't get too distracted and I choose when I want to check things.

My workflow now changed from:

1. I join this interface
   
2. login to my github org SSO
   
3. Reach the Next button until there is no more to sort the list by chronological order
   
4. Go through each Notification, treat it
   
5. Mark it as done
   

to:

1. Go through each Notification, treat it
   
2. Mark it as done
   

Which is a nice reduction of 3 steps, the third one varying with the repo activity.

Enjoy!