Did we make the most of living here?

So we wrote down everything we had experienced across Northern Virginia, Washington, DC, and Maryland. What began as a personal inventory became a surprisingly useful DMV bucket list—and when I shared it with r/nova, the community added even more ideas.

This is not a rigid itinerary or a ranking. It is a farewell tour built from six years of museums, trails, monuments, games, day trips, bakeries, and meals. Whether you are new to the area, hosting visitors, or preparing to leave like us, I hope it helps you notice just how much is within reach.

The short version: Walk the monuments after dark, watch a plane roar over Gravelly Point, spend a day in the free museums, get out on the Potomac, hike west of the Beltway, eat something you cannot easily find elsewhere, and save room for ice cream.

Virginia: views, trails, history, and easy escapes

Northern Virginia is at its best when you move between the Potomac waterfront, small historic towns, and the foothills to the west.

A plane departs Reagan National above Gravelly Point. Photo by Carl Lindberg, Wikimedia Commons, CC BY 2.5.

Along the Potomac

• Mount Vernon and the Mount Vernon Trail: George Washington's estate makes the history tangible; the paved trail turns the riverfront into a long, scenic ride or run back toward Arlington.

• Gravelly Point: Spread out a blanket and wait for arriving or departing planes to pass startlingly close overhead. It is simple, free, and one of the area's most memorable experiences.

• Old Town Alexandria: Wander King Street without a plan, explore the side streets, and finish at the waterfront. It works equally well as an afternoon stroll or a full date night.

• Great Falls: The Virginia overlooks deliver the drama quickly, while the surrounding trails make it easy to extend the visit. Seeing the Potomac squeeze through the rocky gorge never gets old.

• Pohick Bay and Fountainhead Regional Parks: Pohick is an easy place to get onto the water; Fountainhead offers wooded trails, paddling, and a quick sense of escape without driving all the way to the mountains.

Arlington essentials

• Arlington National Cemetery: Give it more time than a rushed stop at the Tomb of the Unknown Soldier. The scale, hills, and quiet make this one of the most moving places in the region.

• US Marine Corps War Memorial and the Netherlands Carillon: The famous flag-raising sculpture, open lawn, carillon, and skyline view make this an excellent sunset combination.

• The old Rosslyn observation deck: The View of DC once offered a remarkable panorama from Central Place Tower. It has since closed to the public, so this one survives on our list with a respectful “RIP.”

History and aviation

• Manassas National Battlefield Park: Walk the fields rather than only stopping at the visitor center. The open landscape helps the history register in a way a display case cannot.

• Steven F. Udvar-Hazy Center: The National Air and Space Museum's enormous companion facility near Dulles is worth the trip for the Space Shuttle Discovery alone—then you notice the Concorde, the SR-71, and hangars full of aircraft.

• Harpers Ferry: Technically in West Virginia, but too close and too good to omit. Historic streets, river views, and trail access make it one of the best all-purpose day trips from Northern Virginia.

Head west

Shenandoah National Park. Photo by the National Park Service, Wikimedia Commons, public domain.

• Sky Meadows State Park: A rewarding option when you want mountain scenery and Appalachian Trail access without committing to the longer drive into Shenandoah.

• Shenandoah National Park: Skyline Drive is the easy introduction, but the real reward is getting onto a trail. We used the park as a gateway to many hikes westward, in every season.

• Leesburg: A walkable historic center, nearby wineries and breweries, and easy access to western Loudoun make it a flexible day trip.

• Meadowlark Botanical Gardens: A slower outing, in the best way. Walk the gardens in bloom or return in winter for the seasonal lights.

• Charlottesville: Farther afield, but manageable for a long day or weekend. The Downtown Mall, nearby vineyards, Blue Ridge setting, and layers of history justify the drive.

Washington, DC: use the access while you have it

DC can be easy to postpone precisely because it is nearby. We learned to treat the city's museums, performances, sports, and public spaces as the extraordinary local amenities they are.

The Tidal Basin during cherry blossom season. Photo by AgnosticPreachersKid, Wikimedia Commons, CC BY-SA 4.0.

Museums worth returning to

The Smithsonian museums are free, which means you do not have to “finish” one in a single exhausting visit. We made time for the National Museum of Natural History, National Museum of American History, National Air and Space Museum, and National Postal Museum. Each rewards repeat visits and a willingness to follow whatever catches your attention.

Outside that core, we explored the International Spy Museum, Smithsonian's National Zoo, United States Botanic Garden, and U.S. National Arboretum. The Arboretum is especially striking when the azaleas bloom, while the Botanic Garden is an excellent winter reset.

The Old Post Office Tower—not to be confused with the Postal Museum—is a wonderfully odd free viewpoint hidden inside the Waldorf Astoria's historic building. It remains one of the city's less obvious ways to look over downtown.

Monuments, landmarks, and seats of government

• The National Mall memorial circuit: The Lincoln, Jefferson, Martin Luther King Jr., and Korean War Veterans memorials all deserve time, not just a drive-by. Add the Vietnam Veterans and World War II memorials if they are not already on your route.

• The Washington Monument: Seeing it from below is one thing; reserving a trip to the top changes your sense of the city's layout.

• The U.S. Capitol and White House: Public tours take advance planning, but living nearby makes that planning far easier than it will ever be after moving away.

• Washington National Cathedral: Even in a city of imposing architecture, the cathedral feels distinct. Explore the interior, grounds, and hilltop setting rather than treating it as a photo stop.

• Cherry blossoms: Yes, peak bloom is crowded. It is also beautiful. Go very early, stay late, or walk through Hains Point for a little more breathing room.

Our favorite twist: revisit the monuments at night. The crowds thin, the stone glows, and familiar landmarks feel unexpectedly dramatic. A sunrise from the Lincoln Memorial—one of the best suggestions from the Reddit thread—is an equally strong finale.

Markets, gardens, and places to wander

• Union Market and Eastern Market: These are different experiences, and both are worth doing. Union Market is a modern food-hall stop; Eastern Market is best paired with a longer Capitol Hill walk.

• Dumbarton Oaks: The museum and formal gardens feel far removed from the busier parts of Georgetown.

• Hillwood Estate, Museum & Gardens: The former home of Marjorie Merriweather Post combines decorative art, Russian imperial objects, and beautifully designed grounds.

• Theodore Roosevelt Island: A shaded, low-key walk in the middle of the Potomac, reached from the Virginia side.

• Hains Point: Walk, run, bike, picnic, or use it as a quieter extension of a Tidal Basin outing.

• The Wharf: Come for the waterfront, a meal, a show, or a boat connection. It is also an ideal starting point for seeing the monuments from the river.

Shows, games, and nights out

• The Kennedy Center: A performance here feels like an occasion. Even without a ticket, the terraces and river views make the building worth exploring.

• Capitals hockey at Capital One Arena: Few things turn downtown into a shared living room as quickly as a hockey night.

• A Nationals game at Nationals Park: Baseball is only part of the experience; arriving along the river and lingering around Navy Yard complete the outing.

• DC United at Audi Field: A more compact stadium and an energetic crowd give soccer matches their own character.

• Echostage: Big electronic shows belong here, but the city also shines at smaller rooms such as the 9:30 Club, The Atlantis, Black Cat, Union Stage, and The Anthem.

• Passport DC and the spring embassy open houses: For a few weekends each year, embassies invite the public inside for food, music, art, and cultural programs. It is one of those events that could happen only in Washington.

Get onto the water

Kayaking on the Potomac changes the scale of the city. Paddle from one of the local boathouses, take a sunset cruise, or ride a water taxi between waterfront neighborhoods. We also made time for National Harbor, including the Gaylord resort and MGM casino—a polished, very different contrast to the historic core of DC.

Maryland: maritime days and a little more history

Maryland gave us some of the easiest escapes from the DC routine: waterfront streets, crabs on paper-covered tables, breweries, and forts looking back across the Potomac.

Annapolis Harbor. Photo by Bernini123, Wikimedia Commons, CC0.

• Annapolis: Walk the harbor and the narrow historic streets, watch boats move through Ego Alley, and let the day unfold without a packed schedule.

• A proper Maryland crab feast: The point is as much the ritual as the food: mallets, Old Bay, brown paper, and no reason to rush.

• Baltimore's Inner Harbor: Use the harbor as a starting point, not the entire itinerary. It connects easily to historic ships, museums, Federal Hill, and nearby neighborhoods.

• Guinness Open Gate Brewery: The Maryland brewery pairs a familiar name with an unexpectedly expansive taproom and grounds.

• Old Ellicott City: Hillside streets, old stone buildings, shops, and the river make it a charming half-day wander.

• Fort Washington Park: The fort's position above the Potomac explains its purpose immediately and provides a quieter historic outing than many downtown sites.

Our DMV food-and-drink memory map

Food may be what we miss most. The region's strength is not one signature dish; it is the ability to move among Ethiopian, Vietnamese, Filipino, Italian, Palestinian, Korean, and American cooking in the span of a few Metro stops or a short drive.

Here are the places that became part of our six years:

• Pizza and Italian: Stracci Pizza, Sfoglina, Filomena, and The Italian Store.

• Casual Northern Virginia favorites: Pho 75, Hot Lola's, Galaxy Hut, and Silver Diner.

• DC and neighborhood staples: Purple Patch, Busboys and Poets, Falafel Inc, Yellow, Call Your Mother, Habesha Market, and Founding Farmers.

• Bakeries and sweets: Levain Bakery, Georgetown Cupcake, Blue Bottle, and Jeni's Ice Cream.

• Beer: Port City Brewing Company and Aslin Beer Company in Northern Virginia and DC.

This is a memory list, not an attempt to name every “best” restaurant. Businesses change, menus evolve, and locations close, so check current details before making a special trip.

What the Reddit thread told us to eat next

The strongest community theme was to lean into food that is unusually good—or unusually easy to find—in the DMV:

• Korean food and Korean barbecue in Annandale, plus Vietnamese food at Eden Center.

• Peruvian chicken, Ethiopian food, and Bolivian salteñas.

• A meal at one of José Andrés's DC restaurants.

• L'Auberge Chez François or 2941 for a farewell dinner in Northern Virginia.

• The Dabney, Albi, or St. Anselm for a celebratory DC meal.

• A burger and Ozzie rolls at a Great American Restaurants location.

• Ben's Chili Bowl for the history and ritual of the original U Street location.

The practical advice was even better: return to the restaurants you will miss. A final meal does not have to be the most acclaimed one. It has to taste like your time here.

The best ideas we had missed

The Reddit comments turned our retrospective into a new shortlist. If we had only a few open weekends left, these would rise to the top:

1. See Great Falls from Maryland and hike the Billy Goat Trail. The canal, boardwalks, and rocky trail provide a completely different experience from the Virginia overlooks. Check current trail conditions before going.

2. Visit the National Portrait Gallery and Renwick Gallery. Both are manageable without devoting an entire day, and both expand the usual Smithsonian rotation.

3. Walk Huntley Meadows Park. Its wetland boardwalk offers an easy wildlife escape surprisingly close to Alexandria.

4. Spend an afternoon in Occoquan or Middleburg. Both reward an unhurried walk; choose Occoquan for the river and compact historic district, or Middleburg for horse-country scenery and nearby vineyards.

5. Hike Old Rag or visit Bears Den Overlook. Old Rag is the bigger physical commitment and may require advance planning; Bears Den supplies a quicker Appalachian Trail payoff.

6. Visit the National Museum of the Marine Corps and the National Museum of the United States Army. These free Virginia museums were two conspicuous gaps in our history list.

7. See the National Archives and Library of Congress. When else can you casually visit the country's founding documents and one of its most spectacular interiors?

8. Ride the C&O Canal Towpath. One ambitious suggestion was to take the train to Harpers Ferry, stay overnight, and bike roughly 60 mostly traffic-free miles back to Georgetown.

9. Go farther east to Assateague or Chincoteague. The wild horses, beaches, and island pace require more driving, but they feel worlds away from the Beltway.

10. Do one final, ridiculous Beltway lap. This was offered partly as a joke. After six years in Northern Virginia traffic, perhaps it is the most authentic farewell of all.

What six years in the DMV taught us

It is impossible to finish this region. There will always be another museum wing, trail, restaurant, performance, historic town, or overlook that you somehow missed.

That is not a failure of planning. It is the gift of living here.

Our list started as a way to check for omissions before moving. Instead, it became a record of how full these six years had been: planes over Gravelly Point, late innings by the Anacostia, quiet walks through gardens, mountain overlooks, crowded cherry blossoms, and far too many good meals to name.

If you live in the DMV, choose one thing from this list that you keep saying you will do “someday.” Put it on the calendar. Proximity is a privilege, but only if you use it.

And if you are leaving, do not spend your final weeks trying to complete everything. Revisit the places that already feel like yours.

This guide grew from my original r/nova farewell post and the generous recommendations that followed. Attraction hours, reservation systems, trail access, and restaurant locations change; check official sites before making a special trip.

KBAI - Knowledge Based Artificial Intelligence

Core Themes

• BFS - Breadth First Search

• DFS - Depth First Search

• A* Search

• Heuristics

The semester long, core project of the class is designing a software agent to solve the Raven’s Progressive Matrices problems. A series of visual tests that measures human’s intelligence. An example is below.

Fall 2023

ML4T - Machine Learning for Trading

Core Themes

• Software in the Stock Market

  • Stock Market Financial Overview

  • Technical Indicators

  • High Frequency Trading (HFT)

• Linear Regression

• Decision Trees - Random Forest Learner

• Reinforcement Learning - Q-Learner/Dyna

The final project in the class combines the prior machine learning techniques. The goal of the final project is to create a machine learning model where-in the rules are dynamically determined based on a trained supervised classification random forest model. The model was then deployed in a hypothetical stock simulation where decisions are made to buy and sell stock depending on the model’s attributes and the subsequent technical indicators.

The screenshot below depicts my trained model, the Strategy Learner, outperforming the portfolio of the Benchmark and Manual Strategy.

The classic iris flower classification example is a great dataset to get your feet wet in Machine Learning!!

Spring 2024

SDP - Software Development Process

Core Themes

• Mobile Application Development (JAVA) - Android Studio

• Git/GitHub

• Blackbox Testing

• Whitebox Testing

  • Path, Statement, Branch Coverage

• Unit Test Generation

The larger group project in this class is developing an application in JAVA, using Android Studio. The loose structure of working on software in a typical SDLC process was something I’m used to in my full time work. However, it was a great learning experience in the context of JAVA and Android Studio.

The whitebox testing assignments were very engaging and reinforced learning about the different coverage techniques.

The screenshot below is the entry panel of the application my team and I developed.

Summer 2024

AI4R - Artificial Intelligence for Robotics

Core Themes

• Histogram Filter

• Kalman Filter

• Particle Filter

• Kinematic Bicycle Model

• PID Controller (Proportional, Integral, Derivative)

• SLAM (Simultaneous Localization and Mapping)

• Search (A-star, Dynamic Programming (value iteration, optimum policy))

All of AI44’s projects are well thought-out and engaging. To complete the projects in full, you must understand the concepts at a fairly deep level. Each project comes with visual GUI interactions of the tasks, involving flying a drone to a certain height or localizing a moving asteroid.

The following gif is taken from the lectures, visualizing particle filters being constantly sampled and eventually through a particle filter, localizing on the target.

The following visual is a demonstration of the Kalman filter. Predictions and measurements are fed into a filter, based on weights of each model, the prediction and measurements overtime converges into the estimated position of the vehicle.

Lastly, extra credit for the class was implementing a Kalman Filter within moving physical hardware. The following video demonstrates my imperfect Kalman Filter robot turning and stopping autonomously based on ultra sonic measurements.

https://www.youtube.com/watch?v=K2woOHnF7-0

Fall 2024

CN - Computer Networks

Core Themes

• The Open Systems Interconnection (OSI) Model

  • 7 Layer Framework

• iBGP/eBGP

• TCP/UDP

• Congestion Control

• PyBGPStream

• BGP Hijacking/Measurement

• Distance Vector Routing

• SDN Firewall

Spring 2025

VGD - Video Game Design & Programming

Core Themes (Unity)

• AI Tracking Agents

• Animation Mecanim

• Game Feel

• Game Physics

• Game Storytelling

The largest aspect of VGD is a group project to design and build a 3D game. Our group created a maze based, reward finding, combat game called Maze Runner. Below is our game trailer.

https://www.youtube.com/watch?v=fKRjrjPJjEE&feature=youtu.be

Spring 2025

AIES - Artificial Intelligence Ethics & Society

Core Themes

• Fairness, Bias, Legality of Data Collection

• Fairness in AL/ML

• Bias Mitigation in AI

  • Disparate Impact

  • Statistical Parity

AIES centered around bias mitigation, particularly around analyzing datasets to uncover fairness, or lack of. The following is a graph I built from a prison dataset, analyzing the percentage of custody statuses by Gender.

For my final paper, I analyzed a piece of research that compared LLM hiring decisions across different LLM models, namely they compared a proprietary in-house supervised trained model with other big generic LLMs like GPT and Claude.

Summer 2025

NLP - Natural Language Processing

Core Themes

• RNN

• Transformers

• Information Retrieval

• Open Domain Question Answering

• Machine Extraction

• Machine Reading

• Machine Translation

  • Early Seq2Seq Models, Phrase Lattices and Transformers

• Privacy & Societal Impacts

  • Differential Privacy, Clipping Gradients

Fall 2025

AI - Artificial Intelligence

Core Themes

• A*

• Decision Trees

• Bayes Net

• Clustering

• Constraint Satisfaction Problems

• Hidden Markov Models

AI is ALL about writing code from scratch! A lot of these algorithms and concepts I was exposed to in prior classes. The coding exercises were very challenging but often felt like busy work without deep insightful learning.

The exams were fun to study for and take, although they felt more like SAT/ACT style word problems than a computer science class.

Spring 2026

CLAW - Computer Law

Core Themes

• Intellectual Property

  • Patents, Copyrights & Trade Secrets

• Acquisitions

• Licensing

• Data Privacy

The lectures were engaging; the professors clearly cared about crafting thoughtful presentations of the material. The class itself is not very difficult and didn't require a ton of hours for a summer class.

I enjoyed the vague, broad requirements of the two big projects i) Acquisition and ii) Code Analysis. In the Acquisition project, we were required to pick two companies, a target and an acquirer. We then had to do proper due diligence on the target company in the context of assessing their intellectual property. In the Code Analysis project we were tasked with analyzing two IDEs and whether or not there was copyright or other intellectual property infringement.

Summer 2026

In this article I want to specifically cover nuances with the Cookiebot GTM template and how to control opt-in vs. opt-out strategies.

Opt-in vs. Opt-out

Opt-in indicates consent is not granted by default upfront. That is, until a user opts-in to tracking, they are not being tracked. In a real word opt-in configuration, a user visits a webpage for the first time, their consent settings are not initially granted by default. They are shown a consent banner and refuse to interact with the consent banner as they navigate the site. During this time, they are not tracked given the opt-in configuration.

Opt-out indicates consent is granted by default upfront. That is, until a user opts-out of tracking, they are being tracked. In a real world opt-out configuration, a user visits a webpage for the first time, their consent settings are initially granted by default. They are shown a consent banner and refuse to interact with the consent banner as they navigate the site. During this time, they are being tracked given the opt-out configuration.

Further Cookie Consent Documentation

Controlling Opt-in vs. Opt-out in GTM

Within the GTM Cookiebot template, there are options to control opt-in vs. opt-out strategies by taking advantage of the Regions parameter. This parameters allow a change in the default consent settings by region. If a region is not listed, all default consent settings are set to Denied.

In the example below, I’ve added US and changed all of the default consent states to Granted. In this instance, US users will be configured to an opt-out policy given they are initially tracked until taking action to opt-out and change their consent settings.

This configuration, or any configuration can be tested and viewed within the GTM debugger. My physical location is the US, in the debugger, the on-page defaults are all granted as modified by the Region parameter.

US IP Example - Initial Site Load:

I connected to a IP via a VPN to a location in Denmark, an EU country. Now, the on-page defaults are Denied and continues to be Denied because I have not taken any action on the consent banner…yet.

Denmark IP Example - Initial Site Load:

Denmark IP Example - NO - Interaction with Consent Banner:

The current state gets populated a few milliseconds after the on-page defaults propagate, specifically for the event below, the gtm.load event. At this stage, the consent banner has been displayed to the user.

Denmark IP Example - Allow All Cookies - Interaction with Consent Banner:

After allowing all cookies via the consent banner, Cookiebot pushes a custom event cookie_consent_update that changes the consent settings all to Granted. This is the dataLayer push event in the above screenshot. Point-forward the consent settings are saved in the user’s browser. This is an important step to understand because this is why all tags that fire would normally fire on page load need to be changed to fire on the custom event of cookie_consent_update

Otherwise in this Denmark scenario, the Gtag or any other tags that fire upon page load/page view would not fire because they would initially be blocked by the default consent settings. Once the user takes action to change settings (via consent banner), the custom event cookie_consent_update is pushed to the dataLayer and appropriate tags can trigger off of that custom event. On subsequent page loads after consent action has been taken, Cookiebot continues to fire a cookie_consent_update which applies the saved settings of the user. This ensures the Gtag continues to fire on subsequent page loads (even when the consent banner ceases to be shown).

More Examples

• US user visits datatribute.com for the first time -> Piwik Pro, GA4 etc is initially triggered and data captured -> User DENYS tracking via consent banner -> User navigates to another page -> Piwik, GA4 etc is NOT triggered.

• US user visits datatribute.com for the first time-> Piwik, GA4 etc is initially triggered and data captured -> User ACCEPTS all tracking via consent banner -> User navigates to another page -> Piwik, GA4 etc is triggered.

• Denmark user visits datatribute.com for the first time -> Piwik, GA4 etc is NOT triggered -> Consent banner is shown -> User ACCEPTS all tracking via banner -> Consent Update event is triggered -> Piwik, GA4 etc is triggered for initial page -> User navigates to another page -> Piwik, GA4 etc is triggered.

These examples behave this way because of the opt-in vs. opt-out policies controlled by the Region parameter.

Other Notes

• Region codes are ISO-3166 A-2 codes. Full list here.

• Regions that share the same settings can be combined to a comma separated string like US,CA,UK,IT

• In the context of regions, Cookiebot checks for the users physical location via IP origination

Custom Dimensions - Page Metadata with Custom JavaScript

Large websites have categories or groupings of pages and sometimes they don't always follow the same ULR pattern, otherwise analyzing by /blog/content-group1/mypost123 would be easy.

There is often a ton of metadata contained within a CMS and little of is exposed directly on the webpage, either visibly to the end-user or in the URL pattern but this data is still useful for analysis!

In GTM, I've implemented custom JavaScript to pickup meta tags in the page's HTML. This Custom JavaScript variable can then be used to add as a custom dimension in the page view tag.

function getPageSubType() {

  return document.querySelector('meta[name="analytics-subtype"]').content;
}

This is another example that is a little more complex but my client wanted to analyze blog post author performance within Google Analytics. This variable grabs onto an element that contains the author's information. There is some logic in this to account for multiple and single-authored blog posts.

function getMeta() {

  authors = document.getElementsByName('cludo:authors');

  if (authors.length == 1) 
      {
      return authors[0].getAttribute('content');
      }

  else 
  {
  authors_string= ''; 
          for (var i =0; i < authors.length; i++)
            {
                if (i != authors.length -1) //if doesn't equals last iteration, add spacer bt authors
                {
                  authors_string += authors[i].getAttribute('content') + ", ";
                }
                  else 
                {
                 authors_string += authors[i].getAttribute('content');
                }

              }
   return authors_string;
  }
}

Lookup Tables & Custom Triggers - Analyze Hotspots

GTM lookup tables take an input and output a result based on what the input matches. It's a fairly simple logic table that could be accomplished in a Custom JavaScript variable but this built-in variable type is much cleaner and easier to read for most folks.

This specific variable takes in a click class upon the user clicking, depending on where they click on the homepage, those specific links are recorded and segmented by 3 different types.

This allowed my client to analyze which featured links on the homepage were the most popular.

This is another example that utilizes a trigger to specifically track clicks on the favorite button of articles. This trigger then fires the favorite_this tag that funnels a custom event to GA4.

Timers - Analyze Time on Page

With the release of GA4, it's trickier to analyze time on page performance as we were all used to having as a default metric in UA.

GTM timer trigger types allow events to fire at certain intervals. For instance, the example below fires on all pages (Regex matches all) every 15 seconds.

You can also couple these triggers types with form fills, purchase funnels and any other interaction where it's insightful to see how long users take to perform an action.

This is just scratching the surface of what GTM can do! Some might question why GTM is needed when web developers could accomplish all of this by sending custom dataLayer events. That is true however, the power in GTM allows fast iteration and deployment of purely analytical initiatives that are decoupled from the main site's deployment. If you want to learn more about GTM let me know!

I got curious if recruiters or hiring managers were even reading my resume, let alone clicking on the links I provided them on my resume. So I got curious.. how can I get concrete visibility into their engagement?

Here is what the top of my resume looks like

I came up with the thought that I could append UTMs to each of these links for different respective companies. I also figured I could do it programmatically instead of creating a new PDF for each company and manually editing the hyperlinks.

First Step -- Create a Company Tracking List

I started a CSV file that looks like this:

It's not too much work to maintain this list, I updated it every time I was getting ready to apply for a job.

Next Step -- Create a Python Script

• Loop over CSV records from the CSV tracker list

• Read my current resume and extract all hyperlinks using the pdfrw library

• Loop over all companies and modify all hyperlinks

• Save a new PDF for each company

import pdfrw
import csv

def get_csv_records(company_file):

    with open(company_file, 'r') as csv_file:
        csv_reader = csv.DictReader(csv_file)
        return [row for row in csv_reader]

def create_pdf(row,resume_base_file):

    # Load the PDF file
    base_pdf = pdfrw.PdfReader(resume_base_file)
    page = base_pdf.pages[0]

    # Create an empty PDF file
    new_pdf = pdfrw.PdfWriter()

    # Check if the page has any hyperlinks
    for annot in page.Annots or []:
        old_url = annot.A.URI
        if old_url == None:
            continue
        # Replace the old path with the new path
        old_url = str(old_url).strip().replace('(','').replace(')','')

        new_url = (
            f'({old_url}\n'
            f'?utm_campaign={row["campaign"]}\n'
            f'&utm_source={row["source"]}\n'
            f'&utm_medium={row["medium"]})\n'
            )
        # Convert the new string for PDF strings
        new_url = pdfrw.objects.pdfstring.PdfString(new_url)
        # Update the hyperlink
        annot.A.URI = new_url
    # Add the modified page to the new PDF file
    new_pdf.addpage(page)
    # Save the modified PDF file
    new_pdf.write(f'Resume_MyName_{row["Company"].strip()}.pdf')


def main():
    resume_base_file = 'resume_base.pdf'
    company_file = 'company_list.csv'

    data = get_csv_records(company_file)

    for row in data:
        create_pdf(row,resume_base_file)

if __name__ == "__main__":
    main()

I won't break down the code too much, it's fairly straightforward consisting of two core functions, reading in the CSV file and creating a new PDF for each record (Company) in the CSV.

Within the create_pdf() function I utilized the pdfrw library to read the base resume file, and create a new page object based on the base resume. Then we can extract all strings from the page using page.Annots , then we can see if any of those strings meet the criteria of a URI (a web address on the resume).

If the string is a URI, I do some light cleanup on the link and then append/construct the UTMs from the CSV record.

Lastly, the PDF is saved as a new PDF. Essentially we're modifying the base resume each time and then saving it under a different name.

Before Executing

After Executing

I can now shoot off customized resumes to all respective companies. The resume companies will receive will have specific UTMs appended to all hyperlinks in my resume, I'll now be able to see if anyone from Blizzard looked at my resume vs. Anduril.

Conclusion

The totally overengineered result is a neat little Google Analytics 4 chart that I can track the analytics on for any user who clicks the link(s).

I'm able to drill into the detail of whom is looking at my resume, for how long, how often and if they navigate elsewhere within my site(s)

If you'd like to set this up on your own, just remember you need a GA4 property on a website that you own and of course, you need access to that property. In my case, I am also appending UTMs to my Github profile but I'll never be able to see those metrics because I don't own that site of course. In contrast, I have full analytics visibility into my Portfolio site and this Hashnode Blog.

Thanks for reading! Feel free to ask questions if you'd like to know more!

GA4 is great until it isn't. There are 2 big issues with standard (free) GA4, especially if you use Looker Studio:

• GA4 API Quota Limits

• 14-Month Data Retention Policy

GA4 API Quota Limits

If your organization accesses GA4 data through Looker Studio you have probably encountered this error:

Accessing GA4's API via Looker Studio is limited by tokens, there is a certain allowable amount. Tokens are consumed each time a chart or visualization is loaded, pivoted or filtered.

There is some caching that is happening behind the scenes but if you are a power user of Looker Studio and or have multiple stakeholders accessing reports, you can very quickly run into these limits, halting your analytic analyses.

Standard GA4 Limits

In other words, a single GA4 property is only allowed 5,000 tokens per hour. Loading 1 page of 5 visualizations with basic complexity in Looker consumes about ~500 tokens. Your token usage will vary greatly with the number of page objects and the complexity of those queries.

14-Month Data Retention Policy

Standard GA4 has a data retention policy of 14 months. That's to say only 14 months' worth of data is stored on your GA4 instance.

After that period, unless exported, the data is deleted forever...

What About Analytics 360..

Upgrading to 360 has several advantages but in regards to these 2 issues:

• All Quotas are increased by 10x (1,750 tokens per hour to 17,500)

• Data Retention is increased from 14 to 50 months

However.. Analytics 360 is a whopping $50,000 per year. So onto another option for most of us..

Solution - High-Level Steps

Exporting GA4 data to BigQuery

• Set up a BigQuery project and dataset.

• Link your GA4 property to your BigQuery project.

• Choose which GA4 data you want to export to BigQuery.

• Schedule regular data exports to keep your data up-to-date.

Running DBT Models

• Create a Cloud DBT account and connect it to your BigQuery project.

• Copy or Write DBT models in SQL to transform your GA4 data into a format that is more conducive to analysis.

• Use DBT to automate the data transformation process.

• Test your DBT models to ensure they are working correctly.

Visualizing with Looker Studio

• Connect Looker Studio to your BigQuery project.

• Create custom dashboards and visualizations to display your data.

Disclaimers

1 Million Event Export Limit

Google allows standard GA4 properties to export 1M events per day, if you are below this threshold, the export process is free.

Costs

There will be incremental storage costs and querying costs associated with Big Query. However, costs should be fairly nominal, a few hundred dollars a month, for your average-sized organization. See Google Cloud's Cost Estimator and my other [future] article about estimating Big Query costs.

Google - Subject to Change

A lot of folks are upset about the token limitation. Google is subject to change its terms and product offerings at any time. They have already increased the token limit once with the initial outcry and I believe their native solution to this, the Extract Data Source, could get better in time.

But.. not a bad idea to get your exports going regardless so you can retain a longer history of your web analytics.

Other Solutions

There are certainly other solutions to GA4's quota limits. Some involve paying another vendor like Power My Analytics, using Looker's Extract layer or transitioning to native GA4 reports.

All are valid solutions. However, unless you pay for 360, there is no other solution to retain your data. With that, adding a DBT layer isn't too much work. If you're already exporting data to Big Query you might as well do something with it.

All that said, I do believe this may not fit everyone's use case or technical aptitude. I plan on doing another article that lays out other options.

Why DBT?

The reason we need to use DBT is because the native export that comes out of GA4 is incredibly nested. The data needs to go through another step to be usable in analysis.

There can be costs associated with DBT, but you can get away with the free tier depending on your needs. The process can all be done within the free DBT tier, the advantages to an upgraded plan are more organizationally related (more user accounts, access control etc.) at least for this process specifically.

Conclusion

We didn't get too into the weeds on setting up DBT or a GA4 export. Those detailed steps deserve another dedicated write-up. However, I hope this enlightened you and at the very least, got you thinking about taking action with your GA4 data.

More to come! Please leave a comment if you found this helpful or would like to know more!

SQL (Structured Query Language) has been around for over 40 years and remains one of the most widely used programming languages for managing and analyzing data. With the rise of big data, data science, and machine learning, some have speculated that SQL may become obsolete. However, the reality is that SQL continues to evolve and adapt to new technologies, making it a valuable skill for anyone working with data. In this article, I'll explore whether SQL will ever die and why it remains relevant today.

First, it's important to understand that SQL is a standard programming language for managing relational databases. It allows users to create, read, update, and delete data from a database using a set of commands that are easy to understand and use. SQL is supported by all major database vendors and can be used with a wide range of programming languages, making it a versatile tool for managing data.

One of the reasons why SQL has remained relevant is its ability to handle structured data. Structured data refers to data that is organized in a consistent and structured format, such as a spreadsheet or a database. While unstructured data, such as social media posts or images, has become increasingly important, structured data remains the foundation of most data-driven applications. SQL excels at managing structured data and can be used to query and manipulate large datasets quickly and efficiently.

Another reason why SQL remains relevant is its ability to integrate with other technologies. SQL is used in conjunction with a wide range of programming languages, including Python, R, and Java. This means that developers can use SQL to manage data and then use other languages to perform analysis or build applications. SQL can also be used in conjunction with big data technologies, such as Hadoop and Spark, making it a valuable tool for managing large-scale data processing.

Finally, SQL continues to evolve to meet the needs of modern data management. New features, such as JSON support and columnar storage, have been added to SQL databases to improve performance and support new data types. Additionally, cloud-based databases, such as Amazon Web Services (AWS) and Microsoft Azure, have made it easier to deploy and manage SQL databases in the cloud.

In conclusion, SQL is not going anywhere anytime soon. While new technologies may emerge that challenge its dominance, SQL remains a versatile and reliable tool for managing structured data. Its ability to integrate with other technologies and its continued evolution ensure that it will remain a valuable skill for anyone working with data. Whether you're a data analyst, a software developer, or a data scientist, learning SQL is a smart investment in your career.

I was scouring my local Toyota websites daily until I built the following solution to automatically scrape the listings and productionize the script with GitHub Actions.

Goal

• Scrape my local Toyota website for all RAV4 listings.

• Send myself the prices of each car via SMS text message, sorted by lowest price to highest.

Purpose

• Monitor the inventory and fluctuating prices; allowing me to jump on a deposit for a car.

Python Procedure

Identifying HTML Values

• First, I needed to find the HTML values I was after on my Toyota dealer's website.

• I inspected the MSRP value on the inventory page and found

  • <div class = "price_value"> $34,889

Requests and Beautiful Soup

Easy enough, the start of the script:

• Sends a basic get request

• Creates a Beautiful Soup object

• Grabs all div elements with the class name of price_value

• De-duplicate the returned items using the set() data structure

def get_bike_items(url):
    #Use the requests library for a simple get request
    response = requests.get(url)

    #Build the bs4 soup object
    html_soup = BeautifulSoup(response.text, 'html.parser')

    #Find all divs with a name of 
    all_cars = html_soup.find_all('div', class_ = 'price_value')
    all_cars = set(all_cars)

Some light cleaning is needed for the returned strings

• Loop through all cars

• Replace new lines and strip white space

• Append back to a new list

• Sort list

• Return the list

Light Cleaning

    cars_text = []
    for cars in all_cars:
        text = cars.text
        care_c = text.replace("\n","").strip()
        cars_text.append(care_c)

    cars = sorted(cars_text)
    return cars

Constructing String Message

I had two URLs to pass into this function, one for Hybrid and one for Regular gasoline RAV4s but I wanted each page to be sent in one text message

• Loop over array of URLs

• The first URL in the list was for Hybrid RAV4s so I added REG to the end of the first item in the array (index 0) and started sms_text with 'HYBRIDS || '

• Join the returned list of strings into one string, separated by ||

  • '||'.join(get_bike_items(url))

urls = ['https://www.sheehytoyotafredericksburg.com/new-toyota-rav4-hybrid-fredericksburg-va',
'https://www.sheehytoyotafredericksburg.com/new-toyota-rav4-fredericksburg-va']

sms_text = 'HYBRIDs || '

for index, url in enumerate(urls):
    if index ==0:
        sms_text += '||'.join(get_bike_items(url)) +'\nREG ||'
    else:
        sms_text += '||'.join(get_bike_items(url))

#Calling the function below to send the SMS 
send(sms_text)

SMS Message

Lastly, I just needed to send the SMS message!

• Below is the function for sending a SMS message. It works by using an email account to send the message to a phone carrier network

import smtplib
carriers = {
    'att':    '@mms.att.net',
    'tmobile':' @tmomail.net',
    'verizon':  '@vtext.com',
    'sprint':   '@page.nextel.com'
}

def send(message):
        # Replace the number with your own, or consider using an argument\dict for multiple people.
    to_number = 'xxxxxxxxxx{}'.format(carriers['xxxxx'])
    auth = ('xxx@gmail.com', 'xxxpass')

    # Establish a secure session with gmail's outgoing SMTP server using your gmail account
    server = smtplib.SMTP( "smtp.gmail.com", 587 )
    server.starttls()
    server.login(auth[0], auth[1])

    # Send text message through SMS gateway of destination number
    server.sendmail( auth[0], to_number, message)

GitHub Actions Automation

After the script was running, I wanted to be able to send these messages to myself automatically twice a day.

GitHub actions performs very well for this type of use case. It's essentially a serverless framework for executing code. I plan to do another article that goes deeper into Actions.

For now, to get the above script running:

• Push code to a private GitHub repository

• Create a .yml file in your repository with the path of .github/workflows/main.yml

The structure of the YAML is fairly straightforward

GitHub Action YAML File

#Name of the Action
name: car_scrape_sms

#On is the trigger of when this will be executed
on: 
#Here I am using a basic cron schedule 
  schedule:
    - cron: '15 14 * * *'
    - cron: '15 2 * * *'

jobs:
  upload:
    #The execution environment of the runner 
    runs-on: ubuntu-latest
    steps:
      # Checking the repository out to the execution env 
      - uses: actions/checkout@v3
      - name: Set up Python 
        # Installs Python on the env 
        uses: actions/setup-python@v3
        with:
          python-version: '3.10'
      - name: Install dependencies
        #CLI arguments to execute on the runner 
        run: |
          pip install requests
          pip install beautifulsoup4
      - name: run main
        run: |
          python main.py

GitHub Action Keywords Explained

• name: is the name of your Action run

• on: is the keyword to trigger your workflow, here I am using a cron schedule

• jobs: starts the main entry into the workflow

  • runs-on: is the platform OS, essentially the container (runner) environment that your job will run on

  • steps: are sections to categorically separate pieces of your Action

    • name: of your step

    • uses: can call other actions

      • Here I am using another action that checkouts my repository to the container environment that this will be executed on. Essentially it copies all of the scripts in the repo to the execution env.

      • I am also setting up python in the container env with actions/setup-python@v3

        • with: is passing a parameter to specify the version of python from that above action

    • run: here we can specify the commands to run on the runner. You can treat this as the command line of the runner.

This barely scratches the surface of the power of GitHub Actions!!

Action Run Example

Below is an example of an Action run for the above scripts.

Output

Finally here is the output that gets sent to my phone

1

cd and cd ..

• Navigate to a directory cd MyFolder

• Navigate back a directory relative to your current path. If current path is /home/user/MyFolder executing cd .. will bring you to /home/user

2

ls and ls -ls -h

• List all contents in current path. ls executed while on path ./MyDirectory will list main.py, profile_pic.jpg and creds.txt

├── MyDirectory
│   ├── main.py
│   └── profile_pic.jpg
    └── creds.txt

• List all contents and it's attributes in current path AND in human-readable (-h) format. ie. main.py is 903 bytes in size using ls -ls -h

-rw-rw-r-- 1 adam adam   0 Dec 20 09:49 creds.txt
-rw-rw-r-- 1 adam adam 903 Dec 20 09:49 main.py
-rw-rw-r-- 1 adam adam   0 Dec 20 09:49 profile_pic.jpg

3

pwd

• Prints the full name of the path in the current working directory. pwd while in MyDirectory would hypothetically print /home/user/MyDirectory/

4

nano

• nano is a terminal file editor, nano does not always come pre-installed with every flavor of Linux. To create and start to edit a brand new text file: nano main.py will create a file called main.py and bring you right into the text editor.

5

cp

• Copy a single file. cp <source> <destinationpath_and_filename>

• To copy main.py and place it in the same directory cp main.py main_copy.py

• To copy main.py to a new directory, first create the directory mkdir NewFolder and copy main into it main.py NewFolder/main_copy.py

I can't quite pin down the initial allure of RuneScape. In the poorest attempt to explain it.. a whole new universe was open to explore, with dangers, treasures and the greatest sense of community.

I didn't know it at the time but RuneScape (and World of Warcraft) was shaping my future. Gielinor was influencing how I now think about wealth accumulation, how I take actions in today's real world stock market, how I excel at my technology job and most importantly how I interact with others given a common goal.

Bots - IT Automation 💻

RuneScape was my first exposure to IT automation, and more specifically, automation that I could clearly see at work, it was tangible. To get 99 Magic, the clearest, easiest path was to cast a High Alchemy spell repeatedly.. or about 198,000 times. 198k mouse clicks that took 3 seconds each... Point is, this was tedious.

I purchased a 'script' back in the day (I had no idea what a script meant as a 10-year-old). A High Alchemy bot that would move my mouse clicker for me as I launched this specific script client. It would randomly move sporadically to throw off the game's bot detections, then continue to cast the spell repeatedly without the need for me to be at my desk.

Now, these are nothing new and well-known nowadays but I was fascinated that these magical scripts could be created to automate the game. As a 10-year-old, I couldn't fathom how in the world someone would create such a thing. Now as a young professional, I'm learning Lua to script WoW add-ons.

In conclusion, bots or automation scripts opened my eyes to the fascinating world of IT automation. In my career, I've always been attracted to automating things and in fact, it's a large portion of my job description. I've created many automation scripts, small things like automatically moving my organization's Confluence pages to Sharepoint to developing full-scale data pipelines that update millions of records daily.

IT Automation - Key Learnings

• Discovering a world where programmable scripts can save a tremendous amount of time and money

Law Runes - Economics ⌛

Brief context -- there's a concept of worlds in Runescape, which are just different servers that your character can log in to. You can easily hop between worlds and of course, all of your gear, money and stats are the same. However, certain worlds were unofficially known for certain events. I believe it was world 66, which was the Law Running world.

In the law running world, with no skills, gear or money, a fresh player could make money simply by running back and forth between a bank and a highly-skilled player crafting law runes, providing that player with a steady stream of raw materials. At a certain Runecrafting level, a player can make double the amount of law runes with the same amount of raw materials. This steady stream of raw materials saved the Runecrafters a TON of time and also made them money, given the ability to craft 2 law runes from 1 pure essence (the raw material).

In exchange for their service of running raw materials back and forth between the bank, law runners were given law runes.

Now... with that context out of the way. These law runners would accumulate law runes and they were looking to sell their amassed law runes in exchange for RuneScape gold.

I was not the first one to exploit this opportunity but I observed that I could buy law runes at a lower price at the bank that these law runners were hustling back and forth between. Then, I would teleport to an area where other players were training their magic and burning law runes to do so. These players training magic did not want to interrupt their gameplay to seek out more law runes. I'm sure you know where this is going -- I then sold those law runes to the players training magic at a slightly higher price than the price I paid at the law running bank.

Economics - Key Learnings

• Buy Low, Sell High

• People are lazy, or more fairly, people are willing to exchange money for time

• Running law runes required no skill or experience. That's great. But given that, you also gained no skills or experience by being a runner; time is better spent building upon skills and experience, even if you take a monetary loss in the short term.

Disclaimer: For my RuneScape fans, this was before the grand exchange, a centralized system to post and purchase goods.

More Reading: Law Running - Cornell Edu

Quests - Requirements ❗

Quests in RuneScape required you to follow quest guides written by third parties. There are usually multiple steps involved, not always high in complexity but you did need to be organized and diligent to be efficient..and in doing so, avoid frustrating yourself.

Following quest guides forced me to sit down, read through the entire guide and THEN come up with a game plan. Gathering the requirements first, reading the guide once again, ensuring I have each step planned out and finally executing the quest.

I like to think this early exposure to reading digital documentation and executing the steps on a platform, albeit a video game, helped train my brain to the very analogous procedures I deal with as an IT professional.

For instance, given a business need to connect to an SFTP and download files on a continuous basis. I need to first gather the requirements.

• Does the SFTP require special credentials, via a VPN or other

• When are the files being dropped?

• Will the files be archived?

• Are the files themselves encrypted with a privacy protocol?

And lastly, I need to read the documentation on the SFTP python library paramiko to utilize the package's function to connect to the SFTP in a secure way.

Quests - Key Learnings

• Requirement gathering is an advantageous skill in the world of software development. Learning how to do this all started with figuring out how many burnt chickens I needed and how much iron ore I need to mine and where to complete a Runescape quest!

• Training myself to thoroughly read through technical documentation at an early age has helped me read through new materials and apply those learnings with speed and quality. Yes, it's arguable that the Recipe for Disaster quest is technical for a 10 year-old :)

I won't elaborate on these last two but for those that have played other MMOs and probably relate with these bullets.

Guilds - Collaboration

The Wilderness - Risk vs. Reward

There is a lot of research pointing to the benefits of video games in our early years. At the very least, I think the argument that video games make teens violent is behind us. Regardless of your position on screen time for children and teens, Runescape, among other video games taught me valuable lessons, ones that I continue to reflect on in my professional and personal life.

However, If you have a 64GB, or larger SD/USB, you will need to force format a FAT32 format type. Both Mac and Windows Disk Utilities usually do not give you the option to format a 64GB card to a FAT32 type. As such, you need to force the FAT32 type via the command line.

Steps:

• Open up a terminal and get a list of your mounted storage devices:

diskutil list

• Locate your SD/USB and take note of the IDENTIFIER

NOTE: This next command will ERASE existing data on your device. Double check you've located the correct device

• Run command:
  • change diskname to the name you want
  • change diskidentifier to your disk listed in the step above

sudo diskutil eraseDisk FAT32 diskname MBRFormat /dev/diskidentifier

• If your SD/USB is mounted at /dev/disk2 and you want to rename the removable storage to 'RPI' your command will be:

sudo diskutil eraseDisk FAT32 RPI MBRFormat /dev/disk2

Now your storage device is ready to flash the Raspberry Pi OS image to the device.

A parent page has an embedded page with a form or other interaction(s) that would be useful to capture the UTM values in conjunction with those interactions.

The Issue:

A child (embedded) page does not, by default, communicate with the parent page in regards to passing information back and forth. Therefore, UTM values are not passed from the parent page to the child page.

Example:

For example, a user lands on the parent page -- https://datatribute.webflow.io/?utm_campaign=hashnode-utm -- and say there's an imaginary embedded page with a form submission on this parent page. The UTM campaign 'hashnode-utm' will not be passed to the form interactions.

This is especially problematic when the form is curated from a marketing platform like Pardot. Pardot can read UTM's directly and again, by default those UTMs from the parent child are not appropriately passed to the child page/form; withholding pertinent information from Pardot's platform.

The Solution:

To pass UTMs to an embedded (child) page, we need to implement a very simple Custom HTML tag in GTM that fires on an 'All Pages' trigger:

<script type="text/javascript">
var iframe = document.getElementById('replace_with_value_of_iframe_id');
iframe.src = iframe.src + window.location.search;
</script>

• Get the element ID of your embedded page by inspecting the iframe on the parent page and search for something akin to iframe id = the value of this field is to be replaced in the above HTML script.

• What this does is instead of passing the default URL of the parent page, it passes the entire URL from the parent page, including any extra parameters (UTMs) to the child page.

• The iframe.src is essentially the child page's URL.

Flask Application

A simple front-end that only took a few lines of code. The three user inputs correspond to the three tickers displayed on the pygame application. A user in my network can go to 192.xx.xx.xx:8082 within my local network to access this light-weight flask application.

Front-End

Flask - How It Works

After a submission, the new tickers get written to a text file. The pygame application periodically checks if a modification has been made to the ticker_files.txt file. If there is a modification, the stock ticker application reads those new tickers and pulls from the yfinance module.

A few normalization and error checking steps:

• First check if the ticker is valid by hitting the yfinance module
• Stripping unnecessary quotes
• Writing to the text file

#!/usr/bin/python3

from flask import Flask, render_template, url_for, request
import yfinance as yf

app = Flask(__name__)

@app.route('/', methods =["GET", "POST"])
def home():
    if request.method == "POST":
        tk1 = request.form.get("tk1")
        tk2 = request.form.get("tk2")
        tk3 = request.form.get("tk3")
        ticker_list = [tk1,tk2,tk3]
        ticker_list_names = []
        ticker_string = ""
        for t in ticker_list:
            t = t.upper()
            ticker_string += t + ","
            t_object = yf.Ticker(t)
            if t_object.info["regularMarketPrice"] == None:
                return "You have submitted an incorrect symbol: "+ t + ", go back and resubmit"

            else:
                ticker_list_names.append(t_object.info["shortName"].split(" ")[0])

        print(ticker_list)
        print(ticker_list_names)
        ticker_string +=  ticker_list_names[0].replace(",","") + ","
        ticker_string +=  ticker_list_names[1].replace(",","") + ","
        ticker_string +=  ticker_list_names[2].replace(",","")

        text_file = open("/home/pi/Desktop/ticker_files.txt","w+")
        text_file.writelines(ticker_string)

        return  "You've submitted " + tk1 + " " + tk2 + " " +tk3


    return render_template("home.html")


def flask_run():
    app.run(host='0.0.0.0',port='8082')

flask_run()

This function is called every other second to check if there was a modification (by timestamp) to the text file. If there was a change, indicating that a user has submitted new tickers, the ticker symbols are returned and are used going forward to call the prices and % changes.

stamp_old = 0

def check_file_change():
    global stamp_old
    stamp_new = os.stat("/home/pi/Desktop/ticker_files.txt").st_mtime
    if stamp_new != stamp_old:
        #file changed
        stamp_old = stamp_new
        with open("/home/pi/Desktop/ticker_files.txt","r") as f:
            lines = f.readlines()
            return lines
    else:
        return 0

Version control in the Business Intelligence world has never really been a thing, at least in my experience. In 2021, Google came out with a, somewhat rudimentary, version control feature in Looker. Although basic, it helps a ton when working with larger reports that need constant, lengthy updates.

You can enable 'publishing' in Looker and switch between Draft vs. Published versions; allowing you to work on your report without the end user seeing it live.

To enable:

• Edit Mode

• File -> Publishing Settings

Once enabled, this will appear at the top of your report:

Google Support Docs:

2. Parameters

Parameters in Looker are similar to parameters in Tableau. Parameters allow dynamic input by the user, allowing real time computations. You can also pass parameter inputs back to custom SQL. Allowing a query like:

SELECT [parameter_to_select]
FROM myBigQuerytable
LIMIT 1;

Another example of the dynamic power of parameters is using Age as an input from the end user. Given the input, you can then do custom calculations, like displaying that person's birth year.

Parameter Setup:

Birth Year calculation: DATETIME_SUB(today(),INTERVAl Age YEAR)

User Input and Birth Year Calculated:

Google Support Docs

3. Page & Report Level Filters

Ever found yourself endlessly applying filters to each scorecard or table on a page? Well you can create a page OR report level filter and by default, it's applied universally to all report or page objects.

Page Level:

Page -> Current Page Settings -> Add a Filter

Report Level:

File -> Report Settings -> Add a Filter

If there are a few objects you don't want the higher level filter applied to, you can choose not to inherit that filter under that object's data properties.

4. Drill Down Metrics

Visualizing hierarchically related dimensions can be tricky. GDS offers a drill down option for related dimensions. Take an Asset Type dimension and an Assest SubType. When properly set up, the end user can drill deeper into hierarchical dimensions.

5. Align Page Objects

Right click while selecting 2 or more objects on the page and align them either vertically or horizontally. For instance, this option in the screenshot below will snap the two tables vertically in line with each other (based on the middle marker of each visual)

This makes formatting scorecards, tables and visualizations incredibly easy. No more dragging a millimeter here, a millimeter there.

Introduction: UTM (Urchin Tracking Module)

What?

A tracking structure that functions by appending parameters (UTM ‘codes’) to the end of a URL that Google Analytics can read automatically

Why?

UTM fields like utm_source=email_newsletter can give deeper insight into reporting and actions taken by certain cohorts and or email marketing campaigns as a whole

How?

UTMs are appended manually and automatically (by certain email marketing platforms). These parameters can be read by Google Analytics and thus, pivotable by other GA metrics. For instance, we could see other actions, such as downloads taken by users who came through the hypothetical UTM source of: utm_source=email_newsletter

Summary:

When UTMs are deployed and built with consistency; they become an added layer of insight. UTM parameters are fields that are appended at the end of a URL. These fields can be read automatically by Google Analytics. UTMs are tied to the user’s browser cookie.

Therefore, the information in the UTM is carried with the user throughout their current and future web session (barring cookie expiration, cache wipe and new UTMs overriding existing ones). As such, this allows analyses to be done on certain cohorts, audiences, and platforms. Ie. business questions such as “are users more engaged on the website that come from organic search or that come from our curated newsletter?”

UTM Structure:

Example UTM:

https://datatribute.webflow.io/?utm_source=newsletter&utm_medium=email&utm_campaign=product_sale

There are five UTM fields that can be utilized. However for most web reporting standards, the first three are the ones that will and should be used the most.

• Medium
• Source
• Campaign
• Content
• Term

Medium (required): utm_medium

Used to identify the channel origination of the link. What channel did the user receive this link from?

examples:

• google
• email
• social

Source (required): utm_source

Subset of Medium at a more granular level. Used to identify the specific source of content, what site or platform did the user receive this link from?

examples:

• facebook
• twitter
• mailchimp
• salesforce

Campaign (required): (utm_campaign)

Used to identify a specific initiative.

examples:

• spring_sale
• product_sale
• june_newsletter

Term: (utm_term)

Used to identify keywords in an ad (can be used for versioning of newsletter)

examples:

• shoes
• womens shoes
• red shoes
• sport shoes

Not commonly used outside of cost per click (cpc) advertising

Content: (utm_content)

Used to differentiate type of ads (can also be used for versioning of newsletter)

examples:

• extended_text
• display
• auto_ad

Not commonly used outside of cost per click (cpc) advertising

Important Note:

There are no hard and fast rules for these parameters, except for medium and source. Medium and source be consistently used and enforced by the common ruleset. However, the other parameters can be independently defined on how to use them based on specific reporting and segmentation needs.

Building and Record Keeping

UTMs are unfortunately very manual to build. Unless links and content are utilizing a software or platform like Mailchimp or Hootsuite where UTMs can be automatically appended to all links.

I recommend keeping track of all UTMs that are built in a spreadsheet.

Implementing

After the URLs are built, the links should be copied and pasted in the platform where you are sending the links out from. For instance, UTM links can be copied into tweets, newsletter content, facebook posts etc.

Hypothetical UTM Use-Case

I’m going to send a curated email to my top 100 customers from my personal gmail account. There’s a special offer on my website that I’m going to link my clients to, directly within my email. I want to know how well (or poorly) the personal email drove sales to this special offer as opposed to another messaging communication.

Create the UTM

https://https://datatribute.webflow.io/?utm_source=newsletter&utm_medium=email&utm_campaign=top_client_personal_email

Paste into my email

Using a hyperlink to ‘hide’ the UTMs: “Check out this special offer!”

View insights in Google Analytics

In GA, I will be able to see, by filtering by this campaign, if this specific messaging communication drove any sales.

UTMs & Custom Event Tracking

In combination with custom event tracking the power in UTMs is gaining a deeper level of insight into user behavior. To unlock this insight, custom events in Google Tag Manager can be set up to track specific behavior for specific audiences or initiatives. For example, did the June newsletter drive more downloads(event tracking) than the July newsletter, if so why and how can we improve the newsletter content to drive incremental growth?

Filtering by UTMs in Google Analytics is simple and easy. After UTMs are appended and a link is clicked - GA handles the rest (funneling, sorting and tying with other data). Any other action the user performs on the site, the UTM is carried with them.

The core focus was of course SQL and Python with some data infrastructure and system design questions. Below is a breakdown of the type of technical questions asked as well sample questions and prompts.

• SQL: 45%
• Python: 35%
• Data Systems Design: 20%

SQL

Verbal:

• Explain the difference between a left join and an inner join
• What's the difference between Union vs. Union All?
• Explain what indexing is and it's purpose in a RDBMS
• When analyzing a slow query, what steps would you take to diagnose the issue?

Written:

Given an order and customer table, answer the following questions:

• Take both order and customer tables and produce the order_amount for all customer_ids
• Take the order table and sum the order_amount for each month and display by order_type. Each order_type should be in its own column.
• Take the order table and generate an average_order_type_amount, by order_type for each row.

Order Table:

Customer Table:

Python

Verbal:

• What's the difference between a list and a tuple?
• Explain what a dictionary is
• In pseudocode, explain a how to encode a string input to $ if the character repeats or to ^ if the character does not repeat.
  • For instance, temperature would encode to $$^^$$^$^$$
  • t repeats twice and thus encoded to $

• What is the output of this function?

  mydata = {'foo': 'foo', 'bar': 'bar'}
  def transform(data):
   local_data = data
   local_data['foo'] = 'baz'
   transform(mydata)
   print(mydata['foo'])
  

Written:

• Given a string as an input, write a function to replace all # hashtags with a - and replace all & ampersands with a +
  • For example, ###&## should return ---+--

• Find the longest consecutive recurrence of an element in a list. Given this list:

  [A ,C ,D, D, D, B, C, C, A, F, A, D, A, A,]
  

• Print dictionary in alphabetic order by first name. Expected output Anthony is in IT, Ben is in HR...

  mydictionary = {'Jess': 'IT', 'Will': 'Sales', 'Anthony': 'IT', 'Ben':
  'HR' }
  

Data Systems Design

Prompt Example:

Be prepared to talk about the architectural design of data infrastructure you have worked on, preferably acted as the lead engineer. Prepare architecture diagrams in advance, or you can draw one live as you explain the design.

• Explain the infrastructure and the principles that led to this architecture
• Explain the reasoning behind different design decisions and alternative solutions
• Explain further extensions of the project for future requests
• What was the overall impact the project had?

Questions:

• How would you design a scalable pipeline to ingest flat files?
• How would you design an API client to interact with a microservice?

Virtual Machines:

Linode

Upon signup, you will receive $100 in credits - you do however have to enter a credit card to have on file. If you're just tinkering around, you should never come close to the limit and should never be billed.

$100 is enough for you spin up a 1 CPU, 25GB Storage and 1GB of RAM Ubuntu VM and have it running 24/7 for 20 months!

Safe to say I'll never run out of credits, as I usually tinker around, then tear it down immediately.

https://www.linode.com/

Low/No Code Web Builders:

As someone who's not an expert on front-end languages, I have a ton of fun learning and playing around with these page builders.

Webflow

The Free Starter plan offers you plenty of features to get started, if you'd like to add your own domain name, embed more custom features, add custom HTML and rid the Webflow white label, you'll have to upgrade.

However, the free plan allowed me to build a few pages, perfect for my needs. I built a portfolio site on Webflow that houses my data visualization gallery and client projects.

The Starter plan only allows two static pages which is kind of a bummer however you can use a little trick to create more. I've added a few CMS Collection pages and use those to add more static pages.

The functionality of this web builder is more detailed and feature rich than a Wix or Squarepsace, something more applicable to superusers who are looking to customize their sites a little more. If you're looking for super easy drag and drop features I'd recommend Wix or Squarespace but Webflow is still super easy, there is just a slight learning curve.

https://webflow.com/

Hashnode

If you're reading this from Hashnode, chances are you have a Hashnode account - I am really enjoying writing articles on this site, it's super user friendly and allows the writer to just go, no fussing around with blocks, margins, picture scaling etc.

If you're reading this elsewhere or on Hashnode and you don't have an account yet. Sign up and get writing!

Nicepage

I've created Python Flask apps that look like they're from the 90s. Nicepage has allowed me to easily design a front end, export it and plop it into my Flask app.

I couldn't tell you the limitations of the free tier of Nicepage because I haven't run into any blockers. The difference for me with Nicepage and Webflow is that I can import and export with Nicepage. Once I export I can delete the project in Nicepage and save it locally, in case I ever want to make any changes.

You can of course also publish through Nicepage but I've used it solely to add a little jazz to my Python applications.

https://nicepage.com/

Databases:

Elephant SQL

Elephant SQL is one of the best services I've come across for messing around with a small db server. It's PostgreSQL and their free version gives you 20MB of data with 5 concurrent connections.

Now, 20MB is not a ton, but if you're just messing around with hooking a db server into your application or want to store personal records, it's perfect. I've personally been building a script on my microcontroller that's reading my houses's temperature and humidity, to batch insert data to my ElephantSQL instance every few hours. 20MB is plenty for this use case.

https://www.elephantsql.com/

MongoDB

MongoDB has a shared cluster that is free FOREVER with 512MB of storage. It's not going to be the fastest thing but it's a great service that can give you a foray into NoSQL, document type storage systems.

Just remember if you spin up an instance - to create a connection to it every so often. They will discontinue your database if it's inactive for a certain period.

https://www.mongodb.com/

Computing:

AWS Lambda

I run several tiny Python scripts on AWS Lambda, which is a serverless computing service offered by AWS. I have a home server but I don't run it 24/7. Lambda allows me to run small automation tasks that cost me $0. This isn't just through the free tier, if your scripts use little memory and don't take long to run, they can be essentially free.. or a fraction of a penny.

https://dashbird.io/lambda-cost-calculator/

Check out my other article to learn more about getting a Python function to run on Lambda:

• Serverless Computing

Heroku

Just created a Django app but don't know where to host and publish it? Heroku has great guides for many supported languages. You can't add a custom domain but to get your app published and usable on the web within an hour, the Heroku free tier is a great option to host a small hobby or test application.

https://devcenter.heroku.com/articles/deploying-python

Productivity

Confluence

I was getting tired of Google Docs, of course there's a hierarchical structure to it with folders but I wanted something easier to manage, edit, add content and organize.

Confluence allowed me to organize my life a bit better than Google Docs. I can create hierarchies through child/parent pages, take notes, drop pictures in without formatting issues, drop in easy code snippets or create a table - it's just 10x easier to use than Google Drive suite - at least for personal productivity and digital life organization.

I'll still use GDrive to host large file dumps but in terms of actual content, my notes, ideas and lists all live on my private Confluence Space.

Notion

I jumped on the Notion bandwagon.. I couldn't get myself to fully transition from Confluence to Notion for personal productivity. However, I did find Notion easier for daily task tracking and pages that I may want to publish to family members or friends. It's great on mobile devices.

I use notion for daily journaling and to organize my book list. The embedded page features is nice to have a 'storefront' of my books, all organized and if I click into one, I've taken notes and given it comments after I've finished reading. I couldn't quite create this in Confluence with the format I wanted.

Learnings Along the Way:

• setting the I/O pin number to '8' was not working and giving a nan reading. Changing to 'D8' however worked.
• I am not able to upload a script while the input pin (D8) is connected to the board. Arduino IDE throws an error that it either can't find the port or it's in use.
  • First I had to disconnect the D8 pin from the board, leaving the ground and power pins in.
    • Then I'm able to upload the script to the microcontroller
    • After uploading, I reconnect the input pin to the microcontroller. I'm able to do this while the board has usb power
• passing true in dht.readTemperature(true); changes the output to Fahrenheit from Celsius

Working code:

#include "DHT.h"
#define DHTTYPE DHT11 //change to DHT22 if needed

DHT dht(D8,DHTTYPE); 
void setup(void)
{

  dht.begin();
  Serial.begin(115200);
  Serial.println("setup");
  delay(1000);//Wait before accessing Sensor

}//end "setup()"

void loop() {

  float h =dht.readHumidity();
  float t =dht.readTemperature(true);

  Serial.print("Current Humidity = ");
  Serial.print(h);
  Serial.print("% ");
  Serial.print("Temperature = ");
  Serial.print(t);
  Serial.println("DHT11 Humidity & temperature Sensor\n\n");
  delay(800);
}

I browse a few finance related subreddits and I was always curious to see how often a certain dollar amount is referenced in the comments. Instead of picking a few arbitrary numbers I decided to gather a few baseline statistics, like the average, median and max dollar amounts that are referenced in the comment section of posts.

‍

For this scenario I pulled from the FIRE, /r/financialindependence subreddit using reddit's API. This subreddit has a daily post titled "Daily Discussion". I figured this would be fairly easy to pull and aggregate from, given the consistency of the daily post titles. In the future it would be interesting to see how these numbers compare with another finance subreddit like /r/personalfinance.

Authenticating

Reddit's API is super straightforward, I created a separate account solely for Reddit projects to keep things clean from my main account. Refer to their documentation to get your credentials.

These are the parameters you'll need to fill in:

#private creds here
reddit = praw.Reddit(
    client_id=,
    client_secret=,
    user_agent=,
    username=,
    password=,
)

Subreddit Search

In my case, the daily FIRE posts have the exact same title. As such, I didn't have to do any tweaking to capture only daily discussion posts

1. Choose subreddit
2. Search posts within that subreddit. This will return an object of submission IDs matching the search
3. Convert into a list of string values

subreddit = reddit.subreddit("financialindependence")

submissions = subreddit.search(
    "Daily FI discussion thread", limit=100
)  # returns submission ids matching this, it's an object(iterator)

submission_list = list(submissions)  # list of string values for each daily id

Creating a Submission Instance

The prior step just gathered the IDs of each submission, now we need to instantiate each submission to gather the metadata associated with each submission.

submission_classes = []
for x in submission_list:
    submission_classes.append(reddit.submission(
        x))

Grabbing Comments

Now we need to loop through each submission and grab the comment body in each submission.

1. Loop through each submission
2. Loop through each comment in the comment body of each submission ‍

This line below comments.replace.more will dictate how far down the comment forest the API needs to crawl. With a limit set to None, this took hours to run. I chose to set the limit at 0 which should just gather the top level comments.

for submissionx in submission_classes:
    submissionx.comments.replace_more(limit=None)
    sub_title = submissionx.title

Given my goal, I only need comments that contain a USD currency symbol to extract my needed data points. To do that I used a regular expression matching '$' anywhere in the comment:

for comment in submissionx.comments.list():
        comment_list.append(comment.body.encode(
            "utf-8"))

Normalizing Comments

Of course there are many ways someone can write money. Such as:

• $20k
• $20k-50k
• $1M
• $1,353
• $50k/year

1. To normalize these comments I first replaced each common shorthand with the respective number of 0s using a dictionary:

replace_dict = {
    'k': '000',
    'K': '000',
    'M': '000000',
    'm': '000000',
    'million': '000000',
    'Million': '000000',
    ',': ''
}

  #If dollar matches isn't blank, regex matched on a dollar comment
        if dollar_matches:
            for key, value in replace_dict.items():
                dollar_matches = [w.replace(key, value) for w in dollar_matches]

            for item in dollar_matches:
                comment_list.append(item)

1. The second biggest normalization was to isolate ranges and 'per' comments such as $50k/year. To do that, I used another regular expression to extract the maximum numerical value in a comment. This does reduce the accuracy if a comment happens to have two separate dollar references but I felt the reduction in noise was better than a few missed numbers.

    normalized_dollars_per_submission = []
    for x in comment_list:
        list_to_int = list(map(int, re.findall(r"\d+", x)))
        normalized_dollars_per_submission.append(max(list_to_int))

As an example, '$50K/year' will be normalized to '50000/year' and then extracted as just '50000' through this process.

Statistics and Writing to CSV

Lastly, gather a few baseline stats using the statistics library and write to a CSV.

average_amt = statistics.mean(all_normalized_comments)
max_amt = max(all_normalized_comments)
median_amt = statistics.median(all_normalized_comments)

ziplist = [average_amt,max_amt,median_amt]

csv_columns_second = [
    'average', 'max', 'median'
]
csv_file = "fire_comments_aggregated.csv"
try:
    with open(csv_file, 'w',newline="") as csvfile:
        writer = csv.writer(csvfile)
        writer.writerow(csv_columns_second)
        writer.writerow(ziplist)

except IOError:
    print("I/O error")

Plotting by Daily Post

This was an aside and not something I originally set out to do, mainly because this analysis doesn't provide many meaningful insights but it's interesting to look at nonetheless! Below is the median dollar amount referenced in all top level comments for the FIRE subreddit daily post, plotted by day for the past 250 days.

df = pd.read_csv('fire_comments.csv', parse_dates=['date'],
                 index_col=['date']) 


fig, ax = plt.subplots(figsize=(10, 10))
plt.xlabel('Date')
plt.ylabel('Median')
plt.suptitle('Median Dollar Amount Commented on FIRE Subreddit Daily Posts')

plt.scatter(df.index.values, df.mediann)


ax.xaxis.set_major_locator(mdates.WeekdayLocator(interval=3))
ax.xaxis.set_major_formatter(DateFormatter("%m/%d"))

plt.show()

Aggregate Conclusion

The following statistics are from the top level normalized comments of the FIRE subreddit daily posts for the period: Aug 11th 2020 - June 16th 2021

‍_

Using Python and AWS Lambda I'm refreshing a Google Data Studio dashboard via Google Sheets with only a few lines of code.

See my other post on scraping my local bike shops eCommerce page for an in-depth look at deployment via AWS Lambda, this article will mainly focus on Google authentication, sheets, data studio and the cryptocompare package.

Create a Google Cloud Project

• If you don't already have a Google Cloud account, login with your Gmail to get started
• Create a New Project or use the default project created for you
• Navigate to API's & Services
• Enable the Google Drive's API service associated with the Gmail you are logged into
• Search for Google Drive and Enable Service

Create a Service Account

• APIs & Services -> Credentials -> Create Credentials -> Service Account
• Name fields however you like, create and SAVE the JSON file
• This JSON file is your authentication file and should be placed in the root folder of your Python file
• Take note of the iam.gserviceaccount.com email - it should also be present in your JSON file ‍

Create a Google Sheet

• Create a blank Google Sheet
• Share the sheet with the service account gmail created in the previous step ‍

Query Crypto Prices - cryptocompare

• pip install cryptocompare
• Plug in a list of coins into a list
• Optional: I have a few obscure coins and not all are available via USDT direct trade, as such I created a second list based on the currency to get historical data only. The get_avg function asks to specify an exchange - this is why you may need a currency specification based on your coins.
• Loop over all coins and call get_price and get_avg
• Append this data to empty lists created before the for loop

coin_list = [
        'BTC', 'XRP', 'NANO', 'OMG', 'ARK', 'SC', 'XRP', 'GNT', 'NEO', 'MIOTA',
        'REQ'
    ]
    currency_list = [
        'USDT', 'USDT', 'USDT', 'USDT', 'BTC', 'BTC', 'BTC', 'BTC', 'BTC',
        'BTC', 'BTC'
    ]
    coin_price_dict = {}
    coin_price = []
    coin_change = []


    for coin, curr in zip(coin_list, currency_list):

        price = '${0:,.2f}'.format(
            cryptocompare.get_price(coin, 'USD')[coin]['USD'])
        hist_price = '{0:,.2f}%'.format(
            cryptocompare.get_avg(coin, currency=curr,
                                  exchange='Binance')['CHANGEPCT24HOUR'])

        coin_price_dict[coin] = [price, hist_price]
        coin_price.append(price)
        coin_change.append(hist_price)

Push Dataframe to Google Sheets

• pip install pygsheets
• Authenticate via secrets.json file (created from service account in previous step) add the columns to the blank dataframe; these columns were built within the for loop in the previous step
• Open sheet via pygsheets, select the first sheet and set the dataframe to begin on the first row and column

    #authorization
    gc = pygsheets.authorize(service_file='secrets.json')

    # Create empty dataframe
    df = pd.DataFrame()

    # Create a column
    df['Coin Name'] = coin_list
    df['Price'] = coin_price
    df['24hr Change'] = coin_change

    #open the google spreadsheet (where 'PY to Gsheet Test' is the name of my sheet)
    sh = gc.open('Crypto Dash')

    #select the first sheet
    wks = sh[0]

    #update the first sheet with df, starting at cell B2.
    wks.set_dataframe(df, (1, 1))

Google Data Studio Dashboard

• Copy this Google Data Studio (GDS) template or create your own!
• Replace the data source with the Google Sheet you created

Deploy Code in AWS Lambda

• Package all dependencies and code into a zip file
• Create a Lambda function and set your desired triggers Important note about pandas and numpy for Lambda - there are a few commons errors that Lambda encounters when using pandas/numpy packages. If you have any errors related to these packages I suggest manually downloading pandas/numpy from pypi.org and plopping them into your zip file. You should download --your python version--manylinux--x86 -- 64, version of the package.
• See my other blog post for a detailed explanation of Lambda functions