Welcome on Ishan Das Sharma

How to Generate Presigned URLs for R2 When Using Cloudflare Workers (And Make Sure They Actually Work)

Wed, 14 Jan 2026 00:00:00 +0000

So you want to let users upload files directly to Cloudflare R2 from their browser? Great idea! Presigned URLs are the way to do it.

You follow the documentation, you generate a presigned URL, and… 403 Forbidden.

Let me show you how to fix all of this.

Step 1: Use the Right Library

Don’t use the official AWS SDK. It doesn’t work in Cloudflare Workers because it needs Node.js APIs that Workers don’t have.

Do use aws4fetch - it’s built specifically for Workers and uses the Web APIs that Workers actually support.

`1`	`npm install aws4fetch`

Step 2: Generate the Presigned URL

Here’s the code that actually works:

import { AwsClient } from 'aws4fetch';

async function generatePresignedUrl(
  key: string,           // e.g., "uploads/myfile.mp3"
  expiresIn: number      // seconds, e.g., 3600 = 1 hour
) {
  // Create the client
  const client = new AwsClient({
    accessKeyId: env.R2_ACCESS_KEY_ID,
    secretAccessKey: env.R2_SECRET_ACCESS_KEY,
    service: 's3',
    region: 'auto',  // R2 uses 'auto'
  });

  // Build the URL
  const r2Url = `https://${env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`;
  const objectUrl = `${r2Url}/my-bucket/${key}?X-Amz-Expires=${expiresIn}`;

  // Sign it - this is the magic part
  const signedRequest = await client.sign(
    new Request(objectUrl, {
      method: 'PUT',
      // DON'T include Content-Type here - more on this below
    }),
    {
      aws: { signQuery: true },
    }
  );

  return signedRequest.url.toString();
}

Step 3: The Content-Type Trap (This One Got Me)

Here’s something that drove me crazy debugging: if you include Content-Type in the signed headers, the browser upload will fail even though curl works fine.

Why? Because when you use signQuery: true with aws4fetch, it only signs the host header. If you try to send other headers from the browser, R2 sees unsigned headers and rejects the request.

Don’t sign Content-Type. Don’t send it manually from the browser either. Just let the browser handle it automatically:

// In your frontend code
const xhr = new XMLHttpRequest();
xhr.open("PUT", presignedUrl);
// Don't do this: xhr.setRequestHeader("Content-Type", file.type);
xhr.send(file);  // Just send the file - browser adds headers automatically

R2 will still store the file with the correct content type. It just won’t validate it against the signature.

Step 4: Configure CORS (The Most Important Part)

This is probably why your upload isn’t working. Even with a perfect presigned URL, R2 blocks browser requests unless you configure CORS.

You need to tell R2 “yes, browsers can upload files here.” Here’s how:

# Create a CORS policy file
cat > cors-policy.json << 'EOF'
{
  "CORSRules": [
    {
      "AllowedOrigins": ["https://your-domain.com"],
      "AllowedMethods": ["GET", "PUT"],
      "AllowedHeaders": ["*"],
      "ExposeHeaders": ["ETag"],
      "MaxAgeSeconds": 3600
    }
  ]
}
EOF

# Apply it to your bucket
aws s3api put-bucket-cors \
  --bucket your-bucket-name \
  --cors-configuration file://cors-policy.json \
  --endpoint-url "https://${R2_ACCOUNT_ID}.r2.cloudflarestorage.com"

During development, you can use "AllowedOrigins": ["*"] to allow all origins. Just remember to lock it down in production!

Step 5: Set Up Your Environment Variables

You need these secrets in your Cloudflare Worker:

# Get these from Cloudflare Dashboard → R2 → Manage R2 API Tokens
npx wrangler secret put R2_ACCESS_KEY_ID
npx wrangler secret put R2_SECRET_ACCESS_KEY
npx wrangler secret put R2_ACCOUNT_ID

Your Worker needs all three to generate presigned URLs.

The Complete Flow

Here’s how it all works together:

Backend (Cloudflare Worker):

// Generate presigned URL when requested
export default {
  async fetch(request, env) {
    const key = `uploads/${crypto.randomUUID()}.mp3`;
    const presignedUrl = await generatePresignedUrl(key, 3600);
    
    return new Response(JSON.stringify({
      uploadUrl: presignedUrl,
      publicUrl: `https://your-domain.com/${key}`
    }));
  }
};

Frontend (Browser):

// 1. Get presigned URL from your API
const response = await fetch('/api/generate-upload-url');
const { uploadUrl, publicUrl } = await response.json();

// 2. Upload file directly to R2
await fetch(uploadUrl, {
  method: 'PUT',
  body: file,
});

// 3. File is now in R2! Use publicUrl to reference it
console.log('File uploaded:', publicUrl);

Testing Your Implementation

Here’s a quick test to see if everything works:

// In browser console on your site
const testFile = new Blob(['test content'], { type: 'text/plain' });

// Get presigned URL from your API
const response = await fetch('/api/generate-upload-url');
const { uploadUrl } = await response.json();

// Try to upload
const uploadResponse = await fetch(uploadUrl, {
  method: 'PUT',
  body: testFile,
});

console.log('Upload result:', uploadResponse.status);
// Should see: 200 (success!)

If you get 200, congrats! If not, check the browser console for CORS errors or network tab for the actual error response.

The Full Picture

Here’s what I ended up with in my production app:

worker/r2-utils.ts:

import { AwsClient } from 'aws4fetch';

export async function generateR2PresignedUrl(
  key: string,
  contentType: string,
  expiresIn: number,
  credentials: {
    R2_ACCESS_KEY_ID: string;
    R2_SECRET_ACCESS_KEY: string;
    R2_ACCOUNT_ID: string;
    R2_BUCKET?: string;
  }
): Promise<string> {
  const client = new AwsClient({
    accessKeyId: credentials.R2_ACCESS_KEY_ID,
    secretAccessKey: credentials.R2_SECRET_ACCESS_KEY,
    service: 's3',
    region: 'auto',
  });

  const bucketName = credentials.R2_BUCKET || 'my-bucket';
  const r2Url = `https://${credentials.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`;
  const objectUrl = `${r2Url}/${bucketName}/${key}?X-Amz-Expires=${expiresIn}`;

  const signedRequest = await client.sign(
    new Request(objectUrl, { method: 'PUT' }),
    { aws: { signQuery: true } }
  );

  return signedRequest.url.toString();
}

Frontend upload component:

async function uploadFile(file: File) {
  // 1. Get presigned URL
  const { presignedUrl, publicUrl } = await generatePresignedUrl({
    filename: file.name,
    contentType: file.type,
  });

  // 2. Upload to R2
  const xhr = new XMLHttpRequest();
  xhr.open("PUT", presignedUrl);
  
  // Track progress
  xhr.upload.onprogress = (e) => {
    if (e.lengthComputable) {
      const percent = (e.loaded / e.total) * 100;
      console.log(`Upload progress: ${percent}%`);
    }
  };

  // Send the file
  xhr.send(file);

  // 3. Wait for completion
  await new Promise((resolve, reject) => {
    xhr.onload = () => xhr.status === 200 ? resolve() : reject();
    xhr.onerror = reject;
  });

  return publicUrl;
}

Final Thoughts

Getting presigned URLs working with Cloudflare Workers was way harder than it should have been. The AWS documentation doesn’t help because Workers aren’t Node.js. Most examples online use the AWS SDK which doesn’t work. Even Cloudflare themselves doesn’t have a document anywhere (that I could find at least!) telling us that we can’t use the AWS SDK (or documenting these gotchas).

But once you know the tricks:

Use aws4fetch
Don’t sign Content-Type
Configure CORS
Let the browser handle headers

…it actually works great! And your users can upload files directly to R2 without killing your Worker.

Hope this saves you the hours of debugging I went through. If you’re still stuck, the issue is probably CORS. It’s almost always CORS.

Good luck!

What I Learned Using Neon Postgres for a Week of Live Traffic

Tue, 09 Dec 2025 00:00:00 +0000

I recently built a SQL practice platform for my DBMS course at IITM BS Degree. About 50-100 students were hitting it simultaneously during peak exam prep. I chose Neon as the backend database, mostly because I was curious about their serverless Postgres offering. Here’s what I learned after a week of actual traffic.

Let me be clear about what this actually tested: student queries don’t hit Neon at all. Those run in WASM on the client side (which I’ll write about in Part 2 because that setup was wild). Neon’s job was simple: serve metadata like problem sets, some aggregate stats, and admin operations. This is basically the lightest possible database workload - maybe 2-3 queries per page load, all reads except when I manually added new problems through TablePlus.

Strictly speaking, this website didn’t even need a database since it could have been entirely pre-rendered. But this project is experimental for me. I used Lovable to scaffold the UI, tried TRPC for the first time, used Bun instead of my usual Python stack, and generally did things quite differently from my usual approach. I’ll detail those decisions in future posts. For now, Neon’s role was serving the application’s metadata and admin operations.

So this isn’t “I ran a high-traffic production database on Neon.” This is “I used Neon as a metadata store for a small educational app.” Keep that context in mind.

What I Was Actually Running

The traffic pattern looked like this:

50-100 concurrent users during exam prep peaks
Each user session: load homepage (one query for aggregate stats - yes, I could have cached this but I was lazy), fetch problem set list (one query), load problem set metadata (one query), load all problems in that set (one query)
Peak query rate: probably 10-20 queries per second, ballpark
Total data volume: laughably small, maybe 50MB including all the problem descriptions
No complex joins, no aggregations, just basic CRUD on small tables

The application server ran on Fly.io (legacy free tier, Singapore region because they don’t let me deploy in India). Neon database also in Singapore since they don’t have an India region. This meant server-to-database latency was negligible, which matters more than client-to-server latency for this architecture.

The Good Parts

Cold starts are legitimately fast. I measured a few wake-ups from scale-to-zero: consistently under 1 second from first query to response. For my use case where users might access the site after a few hours of inactivity, this was completely acceptable. Nobody noticed.

Database provisioning is instant. Supabase takes 2-3 minutes to spin up a new project. Neon takes 5-10 seconds.

Branching is as advertised. I used it to create a dev copy of production data for testing new problem sets. The branch showed up in under 10 seconds, and I could connect to it immediately. I didn’t use any fancy “branch per feature” workflows - just treated it as a really fast database clone. That alone was useful.

Connection quality from TablePlus was surprisingly good. I’m in Kolkata, database is in Singapore, and there was no perceptible lag when browsing tables or running manual queries. For comparison, when I’ve used Supabase instances hosted in Mumbai, there’s a noticeable delay on every interaction in TablePlus. With Neon in Singapore, it felt instant. I don’t know if this is connection pooling magic or better routing or what, but it made admin work much smoother.

The serverless driver handled reconnections gracefully. When the compute scaled down and back up, my application code didn’t notice. No error handling needed, no retry logic, just worked. Compare this to using a direct psql connection: when the compute scales down, you get an “SSL connection has been closed unexpectedly” error, then it auto-reconnects after a few seconds. Functional but ugly.

The Tradeoffs

No India region. For my users (all in India), this added latency. Hard to measure exactly since the app is lightweight, but it’s not ideal. Singapore was acceptable but not optimal.

The serverless driver is noticeably slower than direct Postgres. I didn’t benchmark this rigorously, but subjectively, queries that should return in 10-20ms via direct connection felt like 50-100ms via the serverless driver. This is the cost of the pooling/scaling architecture. For my workload, it didn’t matter. For something latency-sensitive, you’d feel it.

Query metrics reset when compute scales to zero. This is both unexpected, and annoying. You can’t track query patterns over time because the dashboard wipes clean after each scale-down. If you’re trying to optimize slow queries or understand usage patterns, you’re out of luck unless you set up external monitoring.

Connection pooling limit is per-project. The serverless driver supports 10,000 pooled connections, which sounds like a lot until you realize it’s shared across all clients accessing that project. My usage never exceeded a single connection, so I can’t speak to what happens near the limit, but it’s worth knowing the boundary exists.

Thoughts on Pricing

Here’s where Neon’s model gets weird.

The free tier gives you 80 projects, each with 100 compute-unit hours per month. That sounds generous: 80 × 100 = 8,000 CU-hours total.

But there’s a catch: you cannot pool these hours across projects. Each project gets exactly 100 CU-hours, period. You can’t run one project for 800 hours and leave the others idle.

The minimum compute size is 0.25 CU. A month has ~750 hours. Do the math: 100 CU-hours ÷ 0.25 CU = 400 hours maximum. You cannot run even a single free-tier project 24/7. You’d hit the cap halfway through the month.

This would be fine if serverless actually scaled to zero in practice. But here’s what I learned the hard way:

Serverless Databases Require Frontend Discipline

Even one user being online keeps the compute running.

I forgot to disable React Query’s automatic background refetching. The default behavior polls the API every few minutes when the window is focused, and refetches on tab focus. This meant anyone leaving the homepage open - even if they weren’t actively using it - would keep hitting the backend, which kept the database compute alive.

A single browser tab, sitting idle on someone’s screen, was enough to prevent scale-to-zero from ever triggering.

This is the fundamental thing about serverless databases that nobody emphasizes: your database doesn’t scale to zero because it’s idle. It scales to zero when clients stop making requests.

With a traditional VPS running Postgres, your frontend behavior doesn’t affect hosting costs. A $5 VPS costs $5 whether your React app polls every second or stays silent for hours. Serverless inverts this. Every automatic refetch, every “check for updates” feature, every WebSocket keepalive becomes a line item on your infrastructure costs.

If you’re building on serverless infrastructure, audit your frontend’s network behavior:

Disable automatic refetching unless you genuinely need real-time data
Use longer polling intervals (or better: don’t poll at all, use manual refresh)
Think about what happens when users leave tabs open for hours
Consider whether that “sync on window focus” behavior is worth keeping the database awake

The backend being “serverless” is irrelevant if your frontend keeps it awake 24/7.

My Actual Usage

Over the 5-day exam prep period, I used about 25 CU-hours. Compute was active almost 24 hours a day for that period. I stayed comfortably within the 100 CU-hour monthly limit.

But here’s the thing: I spent way too much mental energy worrying about it. Watching the number climb in the Neon dashboard, not knowing when students would be active, wondering if someone had a tab open and was quietly burning through my allocation - the “serverless” cost model added stress that a fixed $5/month VPS wouldn’t have.

For anyone considering Neon for similar usage: 50-100 concurrent users doing light reads will use roughly 5 CU-hours per day if the compute stays mostly active. Scale accordingly.

The Paid Tier Math

Running one database continuously at minimum compute (0.25 CU) would cost about $20/month:

0.25 CU × 750 hours × $0.106/CU-hour ≈ $20

That’s steep for hobby projects. Neon’s marketing targets indie developers and side projects, but the economics don’t support always-on applications at that scale. A $5 DigitalOcean droplet running Postgres would be way cheaper, though you’d handle your own backups, scaling, and operations.

Would I Use It Again?

For this specific use case - a platform serving metadata during a concentrated exam period with real idle gaps - Neon worked well. The cold start performance didn’t hurt user experience, database provisioning was smooth, and the connection quality made development pleasant.

For a project that needs to run continuously, I’d probably look elsewhere. The serverless architecture makes sense for workloads with genuine idle periods, but the reality is that even sporadic traffic prevents scale-to-zero from being useful. The per-project compute caps and pricing make it less attractive for always-on applications than traditional hosting.

Where Neon actually makes sense:

Scheduled jobs that run a few times per day (data processing, reports, maintenance scripts)
Dev/staging environments that only spin up during work hours
Bursty workloads with predictable idle periods (weekly reports, monthly processing)
Projects where you value operational simplicity over cost optimization

Where I’d think twice:

Always-on production apps with steady traffic
Anything latency-sensitive (use direct Postgres, not the serverless driver)
High query volume applications where you need persistent metrics
Projects where you can’t enforce strict frontend discipline around polling/refetching

So yeah, I’d probably use Neon again, but I’d probably reach for SQLite first. The technology works as advertised. The pricing model just doesn’t align well with how real applications behave in the wild.

The platform is live at oppe.dev if you want to check it out. Part 2 will cover the WASM SQL execution setup, why I went that route, and the cursed things I had to do to get Python in WASM to talk to a PGLite database running in the browser.

How to Handle "Bracket" Array Query Parameters (key[]=val) in FastAPI

Thu, 27 Nov 2025 00:00:00 +0000

FastAPI expects array parameters in the standard key=v1&key=v2 format. However, many frontend client generators like my favourite, Orval, default to key[]=v1&key[]=v2.

If you cannot change the client configuration, FastAPI will fail to validate these lists.

The fix is to add a middleware that strips the brackets from keys before the request reaches the routing layer.

The Code

This middleware intercepts the HTTP scope, reconstructs the query_string, and ensures it is re-encoded to bytes to avoid AttributeError: 'str' object has no attribute 'decode' errors in Uvicorn.

from fastapi import Request, FastAPI
from fastapi.datastructures import QueryParams

class FixListQueryParamsMiddleware:
    def __init__(self, app):
        self.app = app

    async def __call__(self, scope, receive, send):
        # Adapted from https://github.com/fastapi/fastapi/discussions/7827#discussioncomment-5144572
        if scope["type"] != "http":
            await self.app(scope, receive, send)
            return

        request = Request(scope)
        
        # Rebuild params: strip "[]" from keys ending with it
        new_items = [
            (key[:-2] if key and key.endswith("[]") else key, value)
            for key, value in request.query_params.multi_items()
        ]
        
        scope["query_string"] = bytes(str(QueryParams(new_items)), "ascii")
        
        await self.app(scope, receive, send)

Usage

app = FastAPI()
app.add_middleware(FixListQueryParamsMiddleware)

@app.get("/users")
def get_users(ids: list[int] = None):
    # Request: GET /users?ids[]=1&ids[]=2
    # Result: ids=[1, 2]
    return ids

Context: See FastAPI Discussion #7827.

How to Force Google Calendar Account Picker When Adding ICS Subscriptions

Wed, 26 Nov 2025 00:00:00 +0000

If you share a link that lets people subscribe to an .ics feed in Google Calendar, you’ve probably seen this pattern:

`1`	`https://calendar.google.com/calendar/render?cid=webcal://your-feed.ics`

It works, but there’s a catch: If the user is logged into multiple Google accounts, Google just picks one. No warning. No choice. Easy way to end up adding the calendar to the wrong account.

Here’s the fix.

Use AccountChooser to Force the Picker

Google has a built-in account picker page. If you send users through it first, they will be asked which account to use before they land in Calendar.

Wrap your calendar URL like this:

`1`	`https://accounts.google.com/AccountChooser?continue=https://calendar.google.com/calendar/render?cid=webcal://your-feed.ics`

That’s it. Clicking this link always shows the account picker, then kicks the user over to the Calendar subscribe screen.

Why This Works

Calendar doesn’t expose a “show picker” flag. AccountChooser does. It pauses the flow, lets the user choose an account, and then continues to whatever URL you gave it.

Make Your Own AccountChooser Link

Here’s a quick tool to generate your own AccountChooser-enabled calendar subscription links:

Your ICS URL:

Your AccountChooser Link:

When to Use It

Anytime you expect users to have more than one Google account (so… basically always), and you don’t want support emails about “Why did this add to my work calendar?”

Easy alias for dealing with Python Virtualenvs

Mon, 17 Nov 2025 00:00:00 +0000

There’s something oddly comforting about a shell function that takes care of boring things for you.

This venv() helper does exactly that.

It checks whether you’re already in an environment, spins up a fresh one if needed, installs your dependencies, and activates it.

To use this venv function, you’ll need to add it to your shell’s configuration file. If you’re using zsh, add the following function to your ~/.zshrc file. If you’re using bash, add it to your ~/.bashrc instead.

Don’t forget to source ~/.zshrc or source ~/.bashrc later!

This requires uv to work

venv() {
  # Exit early if already in a virtual environment
  if [[ -n "$VIRTUAL_ENV" ]]; then
    echo "A virtual environment is already active at: $VIRTUAL_ENV"
    return 1
  fi

  # Create .venv if missing
  if [[ ! -d ".venv" ]]; then
    echo "Creating virtual environment..."
    uv venv || { echo "Failed to create venv"; return 1 }

    # Install dependencies if requirements.txt exists
    if [[ -f "requirements.txt" ]]; then
      echo "Installing dependencies..."
      uv pip install -r requirements.txt || { echo "Dependency install failed"; return 1 }
    else
      echo "No requirements.txt found; skipping installation."
    fi
  fi

  # Activate the environment
  source .venv/bin/activate
}

Of course, if you use uv run for running things anyway, then you don’t need this!

I’m an IIT Madras Student. But to Some, I’m Diluting the Brand.

Fri, 31 Oct 2025 00:00:00 +0000

I have a confession to make.

Before I was a student at IIT Madras, I hated it.

For years, the letters “I-I-T” were a trigger, a symbol of a world I wasn’t welcome in: a club with gates so high I couldn’t even pretend to scale them. It left a bitter taste in my mouth.

This wasn’t an abstract dislike; it was a deeply personal one, forged in the crucible of teenage grief and necessity.

My story isn’t the one you’ve seen in coaching center posters. I first learned to code in 2012, not to crack an exam, but out of raw curiosity, the kind that comes from discovery, not strategy.

That curiosity became a lifeline in 2016. I was 15 when my father passed away, and the predictable path my life was supposed to follow – school, college, career – shattered overnight. While my peers were losing themselves in the labyrinth of H.C. Verma and I.E. Irodov, I was juggling freelance coding projects, trying to help my family stay afloat.

From 2017 to 2020, irony became my daily companion.

I was a software developer at an EdTech company. A company that coached thousands of teenagers for the JEE.

I was, in a sense, a builder of the very dream I had failed to attain. I gave the exam a shot, of course. It felt like a rite of passage.

But my mind, occupied with work, grief, and the weight of adult responsibility, wasn’t wired for that brutal, all-or-nothing, three-hour performance. I failed, and I failed decisively.

The sting of that rejection was real.

It came not from failure itself, but from the quiet pity in relatives’ eyes and the invisible social judgment that comes from missing the one exam our society worships.

To cope, I built a fortress of cynicism. “The IITs are just factories for rote learners,” I’d tell anyone who’d listen. “They don’t value real-world skills.” It was a textbook case of sour grapes: a defense mechanism against a system that had deemed me unworthy.

Life moved on.

And then, I found a different door: the IIT Madras BS Degree in Data Science and Applications. It wasn’t a consolation prize. It was a pragmatic, modern reinvention of higher education. It was a way for people like me to rejoin the conversation.

I enrolled. I was proud.

And then, through Reddit threads, Discord servers, and WhatsApp groups, I learned that to a vocal group of on-campus students, I wasn’t a fellow IITian.

I was an imposter, a “brand diluter.”

This is my story, but it’s bigger than me. It’s about a deep cultural sickness, a phenomenon I call “JEE-ification.” And it’s one that’s beginning to have global consequences.

From “Cracking JEE” To Tarnishing “Made in India”

“JEE-ification” is more than an obsession with one exam. It’s a way of life: the belief that learning is a zero-sum conquest of filters, not a lifelong journey of discovery.

The goal isn’t knowledge; it’s rank. The reward isn’t mastery; it’s entry into an exclusive club. The entire prestige of that club derives not from what’s inside, but from how many people are kept out.

It’s worth acknowledging that the JEE once had real intellectual legitimacy. It was designed as a filter for rigor, to identify those who could reason abstractly under pressure, not to manufacture exclusion.

But over time, that rigor decayed into a fetish for gatekeeping. The metric replaced the mission.

And that mindset has metastasized.

You see it in the explosion of “bhaiya/didi” coding channels on YouTube. They don’t teach the beauty of algorithms or the architecture of systems; they teach “roadmaps.” They sell “crash courses” to “crack FAANG in six months.”

I watch these ‘roadmap’ videos with a sense of mourning.

They are selling a cheap forgery of the very thing that saved me: the joy of discovery. They’ve turned a boundless landscape of intellectual play into a grim, linear treadmill.

It feels like watching someone reduce a grand symphony to a list of notes to memorize for a test.

It’s a vocabulary of shortcuts.

The tragedy of the ‘roadmap coder’ isn’t that they’re “lazy”. It’s that they’ve been trained to see code as a ladder rather than a language.

They don’t ask, “what can I build?” but “what will this get me?”

The most visible symptoms of this culture were the Express.js and Hacktoberfest incidents.

During Hacktoberfest, a global event to encourage open-source contribution, a popular Indian YouTube channel, CodeWithHarry, in a now edited video, urged his followers to participate with the tagline “t-shirt loot lo” (Go loot the t-shirt).

The result was predictable: maintainers of projects worldwide were inundated with low-effort, spammy pull requests.

Shortly after, a similar channel unleashed thousands of followers on Express.js, one of the world’s most critical software projects.

The goal was again credit, not contribution, leading to a flood of pull requests editing README files just to add a name.

These students weren’t malicious; they were doing exactly what they’d been trained to do: optimize for a visible credential, be it a limited-edition t-shirt or a green square on their GitHub profile.

But the consequence was global embarrassment. It reinforced the stereotype of the “Indian developer” as someone who games systems rather than contributes meaningfully. The tragedy is that this reputation harms the countless Indian engineers who do build, innovate, and sustain open-source ecosystems with integrity.

This is the endgame of JEE-ification: a culture that values the badge over the building, and in the process, tarnishes both.

And when your sense of identity has been forged entirely in that fire, when your self-worth comes from having survived that one gate, any attempt to open new doors feels like an existential threat. It feels like a rug-pull.

The Purpose Of A JEE Is Who It Selects

In systems thinking, there’s a powerful principle known as POSIWID: The Purpose Of a System Is What It Does.

It’s a simple but profound idea. You don’t judge a system by its stated goals or the intentions of its designers; you judge it by its actual, observable outcomes.

If we apply this lens to the Joint Entrance Examination, the conclusion is unsettling.

The stated purpose of the JEE is to identify the most promising scientific and engineering minds in the nation. But what does it actually do? Who does it consistently, year after year, select?

It selects for incredible resilience, pattern recognition, and the ability to perform under crushing, single-event pressure. These are not trivial skills. But it also overwhelmingly selects for a specific type of mind and, often, a specific life circumstance.

It selects for solvers of well-defined problems, not finders of ambiguous ones. It rewards optimizers of a known system, not builders of new ones. It filters for those who can afford the luxury of a multi-year, single-minded focus on exam preparation, a luxury my own life story shows is not available to everyone.

And by extension, who does it select against?

It selects against the curious tinkerer who spent their teens building open-source projects instead of memorizing reaction mechanisms. It selects against the early-career professional who discovered a passion for technology only after a family crisis forced them into the workforce. It selects against the brilliant student from a remote town who lacked access to the multi-lakh-rupee Kota ecosystem.

The JEE doesn’t just test for aptitude; it tests for a very specific, privileged, and linear life path.

When we deify the exam, we are implicitly devaluing all other paths to knowledge and competence. This isn’t a selection for broad excellence; it’s a selection for a narrow kind of conformity.

And that is the very engine of JEE-ification. It creates a cohort whose identity is forged by this one shared, exclusionary trial. When your entire sense of achievement is based on having survived that filter, anyone who arrives through a different door doesn’t just feel like a peer; they feel like a bug in the system.

In a developing nation where scarcity and signaling dominate opportunity, the exam became not just a filter for competence, but a proxy for self-worth and employability.

The Author Is A Platypus

I have to acknowledge the irony in all this.

I’ve spent paragraphs critiquing credentialism, while pursuing one of the most recognizable credentials in India.

It’s fair to call that hypocrisy. But it’s also reality.

Because even when you see the gate for what it is, you still feel compelled to walk through it. The world keeps too many keys on the other side.

When I joined the IIT Madras BS program, it wasn’t just for curiosity or intellectual revival.

Part of me wanted legitimacy: to reclaim something I’d been denied.

To prove, if only to myself, that I could belong to the club whose doors once stayed shut.

That’s how deep the conditioning runs.

We don’t just internalize the exam; we internalize the need for permission. Even when we try to unlearn the system, its incentives shape our choices.

You can critique the hierarchy and still crave its approval.

Maybe that’s what growing up in a credential-obsessed culture does to you. Awareness doesn’t grant immunity; it only makes the infection visible.

But perhaps naming the hypocrisy is the first honest step toward escaping it.

Life As A “Brand Diluter”

If people think we’re on “easy mode,” they’re living in a fantasy. The IIT Madras BS degree is not a backdoor.

First, our program has its own brutal meritocracy.

It’s not a one-shot exam, but a continuous, multi-year test of endurance.

I’ve been a professional software engineer for almost a decade, and the foundational Maths 2 course still took me nearly a year (three terms) to clear.

The attrition rate is enormous. Many bright students never make it past the foundation level.

If this is “brand dilution,” it’s the most demanding kind imaginable.

Second, we don’t consume campus resources.

We exist in parallel: separate instructors, separate TAs, separate infrastructure.

Our placement cell operates independently from the on-campus placement ecosystem.

We are not your competition. The conflict exists more in memes and myths than in reality.

Third, our community is our strength.

Our WhatsApp groups are alive at 3 AM with students helping one another: a professional in Bangalore tutoring a teenager in Bihar on statistics.

It’s a microcosm of what democratized higher education can look like. We also know the system isn’t perfect.

But we persist. Because the idea is worth the friction.

The Psychology of Exclusivity

Sociologists would call it status scarcity: when prestige is finite, identity becomes a zero-sum game.

If the IIT brand once meant “I passed an impossible test,” then any alternative path, no matter how rigorous, threatens to dilute the currency of that identity.

The fear is existential, not rational. When the club grows, members mistake expansion for erosion.

But the truth is simpler: excellence is not a finite resource. Widening access doesn’t cheapen quality; it amplifies impact.

The brand of an IIT should come from what it builds, not who it excludes.

An IIT Is For Nation Building

If we’re not a threat, and our program is just as rigorous, why does the “brand dilution” myth persist?

Because somewhere along the way, we forgot what IITs were founded for.

Let’s revisit the source. The mission of IIT Madras commits to (emphasis mine):

the advancement of knowledge through education and research, in both Pure and Applied Science, in Engineering, Social Science and Humanities;

service to the community and nation (which are referred to as Extension activity) through the use of their resources both intellectual and material, particularly through Continuing Education for professionals working in Industry.

That’s not fine print. It’s the institution’s DNA. The BS degree is not an afterthought or a side hustle. It’s the 21st-century expression of that mission.

The IIT system itself was born as a postcolonial project of nation-building modeled on MIT and Soviet technical universities, meant to empower a newly independent India with technological self-sufficiency.

Its purpose was not elitism; it was capacity-building. To educate not the few, but the capable. The BS program extends that legacy into the digital era.

And leaders outside the campus bubble already recognize this. Senior IIT alumni, multinational executives, and civil servants I’ve spoken with don’t see “brand dilution.”

They see strategic national value: a model that discovers hidden talent in small towns, mid-career professionals, and working mothers whom the JEE system overlooks by design.

A Message To The Gatekeepers

To the skeptical on-campus student reading this: I understand you. I used to think like you.

My early cynicism toward IIT was born of exclusion; your defensiveness is born of exclusivity. They are mirror images of the same fear: fear that value must always be protected rather than created.

And here’s the uncomfortable symmetry: your fixation on preserving the “IIT tag” is not so different from the “roadmap coder” obsessed with a FAANG job.

Both see learning as a means to a credential, not a craft. Both conflate gates with greatness.

In fact, this obsession with the entry gate over the actual discipline is so pervasive that many on-campus students in core engineering fields spend their four years preparing for software roles, effectively devaluing their own “brand” and degree in pursuit of the same CS-centric credentialism.

The system has taught you that the entry point matters more than the field of study. This isn’t a failure of students; it’s a failure of the culture that “JEE-ification” has built.

My success does not diminish yours. But your gatekeeping diminishes what the IITs can become.

If we could bridge the gap. If on-campus students and BS learners collaborated on research, startups, and open-source projects, IIT Madras could become not just a symbol of excellence, but a prototype for scalable, inclusive education worldwide. A model where rigor meets reach.

Because the strength of the IIT brand in this century will not be measured by the height of its walls, but by the length of its reach.

Easy Jigsaw Puzzle Embed for Your Website

Tue, 22 Apr 2025 00:00:00 +0000

I wanted to share a small project I’ve been working on - a simple jigsaw puzzle embed that you can add to any website.

What It Does

It’s a straightforward tool that lets you embed customizable jigsaw puzzles into your web pages. You can:

Use any image as the puzzle
Choose the background color
Set the difficulty by adjusting the number of pieces

When someone completes the puzzle, it sends a “puzzleSolved” message that your site can detect.

How to Add It

Just include this iframe in your HTML:

<iframe
  src="https://jigsaw.ishan.page?img=https://example.com/your-image.jpg&bg=ffffff&pieces=25"
  width="600"
  height="400"
  frameborder="0"
>
</iframe>

Detecting When the Puzzle is Solved

Here’s a simple code example to detect when someone completes your puzzle:

window.addEventListener("message", function (event) {
  // The puzzle uses parent.postMessage("puzzleSolved","*") when completed
  if (event.data === "puzzleSolved") {
    console.log("Puzzle completed!");
    // Do something when the puzzle is solved
  }
});

Try It Out

Here’s a live configurator where you can customize your own puzzle embed:

Image URL:

Background Color:

Number of Pieces:

Width (px):

Height (px):

Your Embed Code

Preview:

Credits

This is essentially Dillo’s jigsaw puzzle (https://codepen.io/Dillo/pen/QWKLYab) with very minor adaptations to make it work as an embed. I just added the iframe functionality and message event.

The puzzle is hosted at https://jigsaw.ishan.page if you want to check it out.

Announcing a new chess bot - A0lite-js

Sat, 21 Dec 2024 00:00:00 +0000

I’m excited to share a project I’ve been working on - a TypeScript port of the a0lite chess engine. The original a0lite was created as a minimalist neural network chess engine, designed to be simple and easy to understand rather than maximize playing strength.

You can play against the engine by sending it a challenge here. Please note that the bot can only play 3 simultaneous challenges at a time. It accepts bullet, blitz, and rapid challenges, casual or rated.

What was a0lite?

The original dkappe/a0lite project was a Python implementation of a basic neural MCTS chess engine in just 95 lines of code. Unlike Leela Chess Zero (LC0) which is a full Alpha Zero clone with thousands of lines of C++ code, a0lite took a much simpler approach:

Single-threaded MCTS/UCT search
No complex pruning or tree reuse optimizations
Minimal, readable implementation focused on the core concepts
Designed for experimentation and learning

While the original project has been inactive since November 2020, it provided an excellent foundation for understanding neural network chess engines without the complexity of production systems.

Why TypeScript?

While there is an Emscripten port of LC0 that can run in browsers, it has significant limitations and performance issues. By reimplementing the core concepts in TypeScript, we can create an engine that:

Runs natively in modern browsers without emulation overhead
Takes advantage of WebAssembly and Web Workers for better performance
Has a smaller footprint and faster load times
Integrates smoothly with web technologies and frameworks

What is a0lite-js?

a0lite-js is a TypeScript implementation that builds on the original a0lite’s philosophy of simplicity while adding features needed for practical use. It’s designed to work with LC0 neural networks that have been converted to the ONNX format, making them usable in JavaScript/TypeScript environments.

The engine implements:

UCI (Universal Chess Interface) protocol for compatibility with chess GUIs
Monte Carlo Tree Search (MCTS) with UCT for position evaluation
Neural network inference using ONNX Runtime
Support for both standard chess positions and custom FEN positions

Current Limitations

While the engine plays reasonably well in many positions, it has some significant weaknesses:

Mate Blindness: The engine sometimes misses obvious checkmates, even when they’re just a few moves away. This is partly due to the MCTS implementation not specifically prioritizing mate-seeking lines.
Draw Tendencies: A major issue is the engine’s propensity to fall into threefold repetition in winning positions. Even with a clear advantage, it may choose moves that allow the opponent to force a draw through repetition. This behavior stems from:
- Insufficient exploration of alternative winning lines
- Lack of proper draw-avoidance heuristics in the search
- The neural network’s training data possibly including too many drawn games
Time Management: The current time management system is basic and can lead to suboptimal move choices under time pressure.

Future Development

I’m actively working on addressing these limitations. Key areas of focus include:

Implementing mate detection in the search
Adding draw-avoidance heuristics
Improving time management
Optimizing the MCTS implementation for browser environments
Supporting more UCI options
Exploring WebAssembly for performance-critical components
Adding Web Worker support for background processing

The goal is to create a strong, web-native chess engine that maintains strategic understanding while addressing tactical weaknesses.

Getting Started

You can play against the engine by sending it a challenge here.

To try out a0lite-js on your own computer, you will need to have git, bun and lc0 already installed. After that, do the following steps:

Clone the repository and install dependencies:

1
2
3

git clone https://github.com/sad-pixel/a0lite-js.git
cd a0lite-js
bun install

Convert your LC0 network to ONNX format using:
1

lc0 leela2onnx --input=<path-to-network-file> --output=lc0.onnx
Place the resulting ONNX model in the nets directory.
Run the engine using Bun:
1

bun run index.ts

This will start a0lite-js as a UCI chess engine that can be used with any UCI-compatible chess GUI. The engine can also be integrated directly into web applications.

You can play against the engine by sending it a challenge here. Please note that the bot can only handle 3 challenges at the same time.

Contributing

Contributions are welcome! Feel free to open issues or submit pull requests on GitHub. In particular, help with addressing the mate detection and draw-avoidance issues would be greatly appreciated.

Send Applescript Notifications from your CLI easily with this nifty alias

Thu, 12 Dec 2024 00:00:00 +0000

Have you ever wanted to send desktop notifications from your command line on macOS?

This can be particularly useful when you want to be notified when a long-running process completes. For example, you could append ; notify "Task completed!" to any command.

While there are several ways to do this, I find using AppleScript through the osascript command to be the simplest approach. Here’s a handy shell function that lets you send notifications with just a single command.

To use this notification function, you’ll need to add it to your shell’s configuration file. If you’re using zsh (the default shell on modern macOS), add the following function to your ~/.zshrc file. If you’re using bash, add it to your ~/.bashrc instead.

notify() {
  local message="$1"
  local title="${2:-Notification}" 

  if [[ -z "$message" ]]; then
    echo "Usage: notify <message> [title]"
    return 1
  fi

  osascript -e "display notification \"$message\" with title \"$title\""
}

Usage

With a custom title:

`1`	`notify "hello world!" "Custom Title"`

Without a custom title:

`1`	`notify "hello world!"`

Why FileLinks Didn't Work in Google Colab

Sun, 09 Jun 2024 00:00:00 +0000

If you’ve ever worked on a project in Google Colab and tried to provide download links for files using FileLink, you might have encountered some issues. Specifically, you might have noticed that the links often point to localhost:8080, making them useless for actual downloads. This can be frustrating, especially when you just want a simple way for users to download files generated by your notebook.

So, why does this happen, and what can you do about it?

The Problem with `FileLink` in Google Colab

FileLink is a handy feature in Jupyter Notebooks that creates a hyperlink to a file stored in your notebook’s environment. However, Google Colab runs your notebooks on remote servers, not on your local machine. When you use FileLink in Colab, the generated link points to localhost:8080, which is your local machine’s loopback address. This doesn’t make sense in a remote server context because your local machine has no idea where the Colab server is.

In simpler terms, FileLink is trying to give you a shortcut to a place it can’t actually reach. This results in broken links that won’t help you or your users download the files they need.

Here’s an example of code that doesn’t work well in Google Colab:

import pandas as pd
from IPython.display import FileLink

# Sample DataFrame to save as CSV
df = pd.DataFrame({'Column1': [1, 2, 3], 'Column2': ['A', 'B', 'C']})
result_file_path = 'sample_results.csv'
df.to_csv(result_file_path, index=False)

# Creating a FileLink
display(FileLink(result_file_path))

When you run this code in Google Colab, the link it generates points to localhost:8080, which is not accessible from the Colab environment, making it impossible to download the file.

The Button-Based Solution

Instead of relying on FileLink, which doesn’t work well in Google Colab, we can use buttons that trigger file downloads. Google Colab has built-in support for this through the google.colab library, making it easy to create buttons for downloading files.

To implement this, you can use widgets from ipywidgets to create buttons and attach functions to handle the downloads. Here’s a simple way to do it:

import pandas as pd
import ipywidgets as widgets
from IPython.display import display
from google.colab import files

# Sample DataFrame to save as CSV
df = pd.DataFrame({'Column1': [1, 2, 3], 'Column2': ['A', 'B', 'C']})
result_file_path = 'sample_results.csv'
df.to_csv(result_file_path, index=False)

# Function to display download buttons
def display_download_buttons(result_file_path):
    result_button = widgets.Button(description='Download Results')
    
    def download_result_file(b):
        files.download(result_file_path)
    
    result_button.on_click(download_result_file)
    display(result_button)

# Call the function to display the download button
display_download_buttons(result_file_path)

This code snippet demonstrates how to create a simple button that allows users to download a CSV file generated in the notebook.

If you’re working in Google Colab and need a reliable way for users to download files, avoid FileLink and go for a button-based solution with colab.files. This not only solves the problem of broken links but also enhances the user experience by providing a straightforward and effective way to download files. Happy coding!

When Using `bc`, `scale` Doesn't Round!

Wed, 06 Mar 2024 00:00:00 +0000

Ever stumbled upon a weird problem and couldn’t quite figure out where things went wrong?

I’ve been there, and so has a friend of mine while tackling a HackerRank challenge.

The task seemed simple: take a mathematical expression as input and print out the result with three decimal places.

However, getting there wasn’t as straightforward as expected.

Let’s walk through what happened and how we can apply the lessons learned to similar problems.

The Initial Problem

The challenge required reading an expression from the input and then outputting the result to three decimal places. My friend’s first attempt looked like this:

1
2

read var
echo $(("scale=3; $var" | bc))

However, one test case failed due to being off by 0.001. This tiny discrepancy was a significant clue that the approach to handling decimal precision and rounding needed reevaluation.

Side Note
Although this is a personal style preference, the usual way I would pipe a string into bc would involve echoing the string or using a heredoc, not directly using the $(()) syntax.

A better version might look like this:
1
2
read var
echo "scale=3; $var" | bc

I suggested increasing the scale to 4 (scale=4) to explore the behavior, which led to failing all test cases. It seemed counterintuitive, but the point was to dig deeper into how the bc command processes numbers.

What We Discovered

After running some tests locally (always a good practice before submitting your solution), we noticed something interesting:

echo $input
5+50*3/20 + (19*2)/7
echo "scale=4; $input" | bc
17.9285
echo "scale=3; $input" | bc
17.928
echo "$input" | bc
17
echo "$input" | bc -l
17.92857142857142857142
printf "%.3f\n" `echo "$input" | bc -l`
17.929

The exploration revealed a crucial detail: bc truncates rather than rounds numbers when applying the scale. That’s where the printf function becomes handy.

The Practical Solution

The workaround is to use bc -l for the calculation to ensure full precision and then printf "%.3f\n" to format the output correctly, rounding to three decimal places.

Key Takeaways

Precision Matters: The failure of a test case by just 0.001 should immediately signal that the handling of decimal precision and rounding is critical.
Test Locally: Always run your code locally first (unless it’s impossible of course!). It helps catch errors and understand the problem better.
Know Your Tools: Understanding how commands like bc and printf work is crucial. bc sets the number of decimal places but doesn’t round, while printf can format and round the output.
Experiment and Learn: Don’t be afraid to change your approach. Testing with a scale of 4 revealed how bc handles precision.
Pay Attention to Details: Small discrepancies can point to larger issues. A mismatch of 0.001 isn’t just a minor error; it’s an indicator that your method of handling precision needs adjustment.

When you face a coding challenge, keep these insights in mind.

Sometimes, the solution involves more than just writing the code; it’s about understanding the behavior of the tools at your disposal.

Happy coding! Have you ever faced a similiar problem before? Tell me all about it!

Read This Next: Using AI For Recommending Posts On My Blog

Sun, 03 Mar 2024 00:00:00 +0000

Featured Photo by Javier Allegue Barros.

Imagine landing on a blog where every “Read This Next” suggestion feels like it was handpicked just for you.

That’s the kind of reading journey I want for you on my blog.

I’ve been experimenting, trying to develop a system that not only understands the content of my posts but also why they might resonate with you, the reader. This feature is now live, and I’m going to explore the process.

Why Move Beyond Traditional Recommendation Systems?

Hugo has a pretty good, very configurable keyword and category based recommendation system. With careful selection and tweaking of weights, I’ve been able to have it recommend stuff to my liking so far.

But that’s not what I wanted.

Have you ever finished an article on AI ethics only to be recommended a generic post on “Top 10 AI Innovations”? It’s informative, sure, but it misses the mark on continuing the nuanced conversation you were engaged in.

I wanted to dig deeper. I wanted to create a recommendation that felt almost like a friend was suggesting what you should read next- Related, but also having a wide range of topics.

This required a system capable of understanding content at a near-human level and making connections between seemingly disparate topics based on underlying themes and ideas.

RAG, Text Embeddings and GPT-4

To bridge this gap, I turned to AI, specifically text embeddings and GPT-4. I used LangChain to make interacting with the tools easier, though in future posts I will explore further on using the core modules directly.

Text embeddings allow us to transform written content into numerical vectors, representing the essence of a post in a form that computers can understand and compare. It’s like capturing the soul of an article in numbers. GPT-4, on the other hand, offers an advanced understanding of language, capable of generating summaries, and even explaining why one article relates to another in a human-like manner.

For instance, an article on the ethical considerations of AI could lead you to a seemingly unrelated post on philosophy, connected by the underlying theme of ethics and technology.

This is an application of the well known RAG (Retrieval Augmented Generation) pattern.

Summarizing Posts

The process starts off when a summary is generated for each blog post using LangChain and OpenAI’s GPT-4 model.

The script employs a map-reduce approach where each document (part of a blog post) is first summarized individually (map step), and then these section wise summaries are further condensed (reduce step).

from langchain.chains import (LLMChain, MapReduceDocumentsChain,
                              ReduceDocumentsChain)
from langchain.chains.combine_documents.stuff import StuffDocumentsChain
from langchain.prompts import PromptTemplate
from langchain.text_splitter import TokenTextSplitter
from langchain_community.document_loaders import UnstructuredMarkdownLoader
from langchain_openai.chat_models import ChatOpenAI

map_template = """Write a concise summary of the following content:

{content}

Summary:
"""
map_prompt = PromptTemplate.from_template(map_template)
llm = ChatOpenAI(model="gpt-4-turbo-preview")
map_chain = LLMChain(prompt=map_prompt, llm=llm)

reduce_template = """The following is set of summaries:

{doc_summaries}

Summarize the above summaries with all the key details
Summary:"""
reduce_prompt = PromptTemplate.from_template(reduce_template)
reduce_chain = LLMChain(prompt=reduce_prompt, llm=llm)
stuff_chain = StuffDocumentsChain(
    llm_chain=reduce_chain, document_variable_name="doc_summaries")

reduce_chain = ReduceDocumentsChain(
    combine_documents_chain=stuff_chain,
)

map_reduce_chain = MapReduceDocumentsChain(
    llm_chain=map_chain,
    document_variable_name="content",
    reduce_documents_chain=reduce_chain
)

post = "./post/name-of-post/index.md"
loader = UnstructuredMarkdownLoader(post)
docs = loader.load()

splitter = TokenTextSplitter(chunk_size=2000)
split_docs = splitter.split_documents(docs)

summary = map_reduce_chain.invoke(split_docs)
print(summary["output_text"])

Embedding and Storing

The generated summaries are then processed through a text embedding model. This converts the summaries into a high-dimensional space where similar contents are positioned closer together, facilitating the comparison of thematic and semantic similarities between posts.

These embeddings are stored in ChromaDB, a vector database optimized for fast similarity searches.

import json

from langchain.embeddings import CacheBackedEmbeddings
from langchain.storage import LocalFileStore
from langchain_community.vectorstores import Chroma
from langchain_core.documents import Document
from langchain_openai import OpenAIEmbeddings

summaries = json.loads(open("sum.json", "r").read())

underlying_embeddings = OpenAIEmbeddings(model="text-embedding-3-large")

store = LocalFileStore("./cache/")

cached_embedder = CacheBackedEmbeddings.from_bytes_store(
    underlying_embeddings, store, namespace=underlying_embeddings.model
)

docs = []
for k in summaries.keys():
    sum_source = k.split("/")[2]
    sum_page = summaries[k]
    doc = Document(page_content=sum_page, metadata={'source': sum_source})
    docs.append(doc)

db = Chroma.from_documents(docs, cached_embedder)

To find similar posts, we query ChromaDB with the embedding of the current post, retrieving other posts with the closest embeddings. This step uses Maximal Marginal Relevance, an algorithm that considers both similarity and diversity, making sure that recommendations are both relevant and varied.

For each recommended post, GPT-4 is used to generate an engaging, brief explanation of why it might be of interest, based on the thematic connections identified through the embeddings.

This step adds a personal touch to the recommendations, making them feel more curated and thoughtful.

from collections import defaultdict

from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate
from langchain_openai import ChatOpenAI

relateds = defaultdict(list)
rec_template = """You are an intelligent AI-powered content recommendation assistant. 

Here's a summary of the article that the reader has just read:
{summary_a}

Here's a summary of the next article recommended for the reader:
{summary_b}

Now, write why the reader might find it interesting based on the contents of both the articles. 
Don't say things like "Given", "Dive in", "Elevate" etc.
Always start with a attention grabbing, relevant and catchy hook.
Your answer should explain in no more than one sentence using clear, concise, conversational language.
Don't be superfluous, use superlatives or exaggerate. Always be practical. 
Always refer to the reader as "you" and the author in the first person.
"""

llm = ChatOpenAI(model="gpt-4-turbo-preview", temperature=0.2)
rec_prompt = PromptTemplate.from_template(rec_template)
rec_chain = LLMChain(prompt=rec_prompt, llm=llm)
for k in summaries.keys():
    print("For: ", k)
    sum_source = k.split("/")[2]
    sum_page = summaries[k]
    found_docs = db.max_marginal_relevance_search(
        sum_page, k=3, fetch_k=len(summaries.keys()) - 1,
        filter={"source": {"$ne": sum_source}}
    )
    for i, doc in enumerate(found_docs):
        result = rec_chain.invoke({"summary_a": sum_page, "summary_b": doc.page_content})
        relateds[k].append({"slug": f'post/{doc.metadata["source"]}', "why": result["text"]})

print(relateds)

Putting It All Together

Finally, now that I have the list of related posts, I’m going to put them in a next.json content file next to each post’s index.md.

{
    "read_next": [
        {
            "slug": "post/2023-07-09-programming-with-the-grain",
            "why": "You might dig the next article because it switches gears from the nifty math trick you just explored to the practical world of programming, showing you how to ride the wave of a language's inherent features for smoother coding. It's like applying a smart formula but in the realm of coding, making your work not just correct but elegantly efficient."
        },
        {
            "slug": "post/2023-06-20-facebook-graph-leads-between",
            "why": "The reader, having just explored a mathematical trick for estimating digits in powers of 2, might find the next article intriguing as it shifts focus from theoretical math to practical application, demonstrating how to leverage Python and the Facebook Graph API for data retrieval tasks. This presents an opportunity to see Python's versatility in action, from performing mathematical computations to interacting with web APIs."
        },
        {
            "slug": "post/2021-09-18-pem-linux",
            "why": "If you enjoyed learning a neat mathematical trick for estimating digits in powers of two, you might find diving into the practical world of cryptography with RSA key generation equally fascinating. It's a cool way to see how mathematical concepts can be applied to enhance security in computing tasks like setting up secure, passwordless connections."
        }
    ]
}

Finally, I create a shortcode to display the recommendations inside the post in layouts/shortcodes/read-next.html:

{{ $dataFile := "next.json" }}
{{ range .Page.Resources.Match $dataFile }}
{{ $data := .Content | transform.Unmarshal }}
# Read Next
I'm running [an experiment](/blog/read-this-next-embeddings-llm-rag/) for better content recommendations. These are the 3 posts that are most likely to be interesting for you:

{{ range $data.read_next }}
{{ $page := site.GetPage .slug }}
{{ if $page }}
- [{{ $page.Title }}]({{ $page.RelPermalink }})  
    {{ .why | markdownify }}
{{ else }}
{{/* Handle the case where the page is not found */}}
{{ printf "Recommended reading not found for slug or filename: '%s'" .slug }}
{{ end }}
{{ end }}
{{ end }}

And here’s the result:

Calculating The Number Of Digits In A Power Of 2

Sun, 25 Feb 2024 00:00:00 +0000

I want to share something cool I learned about logarithms. If you’ve found logarithms a bit tricky, like I did, you’ll love this trick!

It’s a neat way to see logarithms in action and made things click for me.

Someone asked me how I would go about approximating the number of digits in $2^{32}$.

I found this discussion on StackExchange. To quote the top answer:

A general formula for the number of digits of any power of 2 can be found using the value of the base-10 logarithm of 2.

The number of digits in $2^k$ is
$1 + \lfloor k \cdot \log_{10}(2) \rfloor$

The trick uses a fundamental property of logarithms and shows how they relate to the scale of numbers in different bases, particularly base 10 (since we’re interested in the number of digits). Let’s try to break it down:

The logarithm of a number is the exponent to which another fixed number, the base, must be raised to produce that number.

In this case, we’re dealing with $log_{10}$, which tells us how many times we must multiply 10 to get our number.

Given $2^k$, where $k$ is an integer, we want to find out how many digits this number has.

To do this, we can take the base 10 logarithm of $2^k$, expressed as $\log_{10}(2^k)$.

Using the property of logarithms that allows us to bring the exponent to the front, we get:

$\log_{10}(2^k) = k \cdot \log_{10}(2)$

The value of $\log_{10}(2)$ is approximately $0.30$. This tells us how many times 10 must be multiplied to reach the base 2.

When we multiply this value by $k$, we get a sense of how large $2^k$ is in terms of its place in the base-10 system.

The resulting value $k \cdot \log_{10}(2)$ gives us a decimal number that correlates with the magnitude of $2^k$ in base 10.

However, this doesn’t directly tell us the number of digits. We still need to floor it and add 1.

The logarithm gives us the “order of magnitude” of our number in base 10.

For example, $\log_{10}(100) = 2$, meaning $10^2 = 100$ has 3 digits.

The log tells us it’s in the hundreds place, but we know that anything in the hundreds (from 100 to 999) has 3 digits.

By adding 1 to the result, we essentially round up from this “order of magnitude” to the actual number of digits.

We have to take the floor because the result isn’t always clean.

Consider the cases of 553 and 999:

$\log_{10}(553) = 2.472$
$\lfloor\log_{10}(553)\rfloor = 2$
$\lfloor\log_{10}(553)\rfloor + 1 = 3$

$\log_{10}(999) = 2.999$
$\lfloor\log_{10}(999)\rfloor = 2$
$\lfloor\log_{10}(999)\rfloor + 1 = 3$

This works because the number of digits is always one more than its logarithm’s floor value for numbers in base 10.

The formula $(1 + k \cdot \log_{10}(2))$ therefore calculates the number of digits in $2^k$ by using the log to find its order of magnitude in base 10, and then adjusting for the fact that the number of digits is one more than that magnitude.

Let’s try to verify the result:

We’ll calculate the number of digits in two ways using a Python script, with the first method being the formula, and the other by converting $2^k$ to a string and getting its length. Finally we’ll compare the results and note the differences.

import math

results = [(k, 1 + math.floor(k * math.log10(2)), len(str(2**k))) for k in range(1, 501)]
mismatches = [result for result in results if result[1] != result[2]]
print(mismatches[:1])
# []

This verification code loops over $k$ from 1 to 500. Since there is nothing in the mismatches array, it shows that the formula $1 + \lfloor k \cdot \log_{10}(2) \rfloor$ correctly gives the number of digits in $2^k$ for every value of $k$ in this range.

The trick isn’t limited just to powers of 2 - you can use any number in the log.

Therefore, a more general formula for finding the number of digits in $x^k$ would be:
$1 + \lfloor k \cdot \log_{10}(x) \rfloor$

Blogging with Jupyter Notebooks in Hugo

Fri, 15 Dec 2023 00:00:00 +0000

Featured Photo by Simon Berger

I’ve realized something important about myself lately: I need to make it easy for me to publish my stuff. If I have to do a lot of steps to get my text online, I lose motivation and give up. That’s why I switched from my custom PHP site to a hugo site with a ready-made theme on Cloudflare pages. Now I just have to git push and it’s done.

I used to hate working with Jupyter notebooks because they required me to open a browser and deal with all the hassle of web interfaces. But ever since VS Code added support for .ipynb files, I’ve changed my mind.

Now I can easily play around with code and data, and see the results in real time. It’s great for learning new things.

One cool feature of Jupyter notebooks is that they can have markdown cells, where you can write text with formatting. That got me thinking: what if I could use Jupyter notebooks to write blog posts and then render them in hugo? That would be awesome, right?

I present to you:

`1`	`{{ % ipynb "simple.ipynb" % }}`

The rest of this post is written in the Jupyter notebook.

🗒️ Download (simple.ipynb)

This Is A Jupyter Notebook

It has a markdown starting section, that’s quite neat!
Let’s try to create a Python code block next:

2 + 2

Output:

Alright, that seems to have gone well! Note how this second section is sandwiched in the middle of two code blocks.

The next code block uses print to send the output. The reason for this will be important later!

`1`	`print("This is a message!")`

Output:

`1`	`This is a message!`

Thinking about Options

Let’s take a moment and evaluate:

Hugo Page Resources can be used to attach a resource to a post
A Jupyter Notebook is actually just a JSON file.
Cursory googling shows people trying various options, including rendering:
- hugo_jupyter - Turns “notebook” into it’s own type of post. Frontmatter goes inside notebooks
- nb2hugo - Similiar approach as above
- Hugo Blox - Some custom themes take the same approach above
- Forum Topic discussing Jupyter Support
- Skeptric on the nb2hugo workflow

Looking Inside Notebooks

Let’s take a look inside this notebook itself. Since it’s a JSON file, we can use jq and the keys function to start exploring:

`1`	`!jq -M 'keys' simple.ipynb`

Output:

[
  "cells",
  "metadata",
  "nbformat",
  "nbformat_minor"
]

Remember how I mentioned that the fact I used print will be relevant later? It’s coming soon. I promise. Let’s take a look at the first 4 cells of the notebook:

`1`	`!jq -M '.cells[:4]' simple.ipynb`

Output:

[
  {
    "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
      "## This Is An Example Jupyter Notebook\n",
      "It has a markdown starting section, that's quite neat!   \n",
      "Let's try to create a Python code block next:"
    ]
  },
  {
    "cell_type": "code",
    "execution_count": 26,
    "metadata": {},
    "outputs": [
      {
        "data": {
          "text/plain": [
            "4"
          ]
        },
        "execution_count": 26,
        "metadata": {},
        "output_type": "execute_result"
      }
    ],
    "source": [
      "2 + 2"
    ]
  },
  {
    "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
      "Alright, that seems to have gone well! Note how this second section is sandwiched in the middle of two code blocks.\n",
      "\n",
      "The next code block uses `print` to send the output. The reason for this will be important later!"
    ]
  },
  {
    "cell_type": "code",
    "execution_count": 27,
    "metadata": {},
    "outputs": [
      {
        "name": "stdout",
        "output_type": "stream",
        "text": [
          "This is a message!\n"
        ]
      }
    ],
    "source": [
      "print(\"This is a message!\")"
    ]
  }
]

Observations

Every object has a cell_type which is either markdown or code
- All cells have a source. Source is as array of strings containing the cell’s source code line by line.
- Cells which have been executed have an outputs
While outputs is always an array, the contents differ:
- When it’s a normal Python evaluation, the output_type is execution_result.
  In this case, we have to get the data from the outputs[].data which is a object containing keys corresponding to the mime type and value as an array containing the values.
- When it’s using print, the output_type is stream.
  In this case, the outputs[].text is the array of strings we are interested in.

But What About Images?

Fair point! We haven’t encountered any in this notebook yet. Let’s change that by copying in one of the matplotlib examples:

import matplotlib.pyplot as plt

fig, ax = plt.subplots()

fruits = ['apple', 'blueberry', 'cherry', 'orange']
counts = [40, 100, 30, 55]
bar_labels = ['red', 'blue', '_red', 'orange']
bar_colors = ['tab:red', 'tab:blue', 'tab:red', 'tab:orange']

ax.bar(fruits, counts, label=bar_labels, color=bar_colors)

ax.set_ylabel('fruit supply')
ax.set_title('Fruit supply by kind and color')
ax.legend(title='Fruit color')

plt.show()

Output:

`1`	`<Figure size 432x288 with 1 Axes>`

Let’s take a look inside this last cell:

`1`	`!jq -M '.cells[-3].outputs[].data \| keys' simple.ipynb`

Output:

[
  "image/png",
  "text/plain"
]

Let’s take a look at what’s inside the image:

`1`	`!jq -M '.cells[-3].outputs[].data["image/png"][:32]' simple.ipynb`

Output:

`1`	`"iVBORw0KGgoAAAANSUhEUgAAAYUAAAEI"`

It’s a base64 encoded png! That makes our life easier.

Now, with all the observations so far, we now have enough information to create the shortcode

The `ipynb` Shortcode

Why a Shortcode?

Most of the previous approaches use layouts, but I have gone with a shortcode instead. Using this approach lets me keep the frontmatter in my post markdown file.

The main benefit of this is that I can have multiple notebooks inside a post if I want.

Copy Me!

The below code goes in layouts/shortcodes/ipynb.html

{{ $notebook := .Get 0 }}
{{ range .Page.Resources.Match $notebook }} 
<a href="{{.RelPermalink}}" download>🗒️ Download ({{ $notebook }})</a>
{{ $content := .Content | transform.Unmarshal }}
{{ $cells := $content.cells }}
{{ range $cells }}
<!-- For Markdown Cells -->
{{ if eq .cell_type "markdown" }}
{{ range .source }}{{ . | safeHTML }}{{ end }}
{{ end }}

<!-- For Code Cells -->
{{ if eq .cell_type "code" }}
```python
{{ range .source }}{{ . | safeHTML }}{{ end }}
```
{{ if gt (len .outputs) 0 }}
Output:
{{ range .outputs }}
{{ if .text }}
```
{{ range .text }}{{ . | safeHTML}}{{ end }}
```
{{ end }}
{{ range $type, $data := .data }}
{{ if strings.Contains $type "image" }}<img src="data:{{ $type }};base64,{{ $data | safeHTML }}">{{ end }}
{{ if strings.Contains $type "text" }}
```
{{ range $data }}{{ . | safeHTML }}{{ end }}
```
{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}

How does it work?

This Hugo shortcode is designed to embed a Jupyter notebook into a Hugo website by parsing and displaying its content. It processes the Jupyter notebook content, separating Markdown and Code cells, rendering them accordingly on the Hugo page. It also handles outputs generated by code cells within the notebook.

Let’s try to break down how it works:

{{ $notebook := .Get 0 }}: This line gets the parameter passed to the shortcode.
{{ range .Page.Resources.Match $notebook }}: This initiates a loop over the resources associated with the current page that match the provided notebook name.
<a href="{{.RelPermalink}}" download>🗒️ Download ({{ $notebook }})</a>: This generates a download link for the notebook file. The .RelPermalink is the relative path to the notebook file.
{{ $content := .Content | transform.Unmarshal }}: This extracts the content of the notebook and stores it in the $content variable.
{{ $cells := $content.cells }}: This extracts the cells (sections) of the notebook and stores them in the $cells variable.
{{ if eq .cell_type "markdown" }}: This checks if a cell is of type “markdown.”
{{ range .source }}{{ . | safeHTML }}{{ end }}: If the cell is a markdown type, it iterates through its content and renders it as HTML using safeHTML.
{{ if eq .cell_type "code" }}: This checks if a cell is of type “code.”
python ... : This wraps code cells in a Python code block for display.
{{ range .source }}{{ . | safeHTML }}{{ end }}: Renders the content of the code cell as HTML using safeHTML.
{{ if gt (len .outputs) 0 }}: Checks if the code cell has any outputs.
Renders outputs:
- Text Outputs: Renders text outputs of the code cell.
- Image Outputs: Renders image outputs encoded in base64 format.
- Other Outputs: Handles other types of outputs (text or image) based on their data.

Limitations and Closing Thoughts

While the shortcode attempts to render Markdown and Code cells, it might not cover all possible cell types or formatting nuances within a Jupyter notebook.
Inline embedded images may be bad for performance as they aren’t being lazy loaded
This approach skips over a lot of the extra features which Jupyter notebooks have. I couldn’t even tell you what they are because I don’t know myself!
While this is perfect for my workflow, and I’m excited by how much easier it makes for me to get more posts out by removing friction, I recognize that this shortcode/approach may not be for everyone.
There is official documentation for the .ipynb format, but I didn’t really look at it. Someday in the future I might have to do that.

I hope you enjoyed reading this post. Have you experienced similiar things about the importance of removing friction in your workflow? Do you have any stories of things you automated to that end? Tell me all about it!

I’ve finally started a newsletter for those who wanted to receive updates over email.
You can subscribe below, or check out the first issue here

Hello, World!

Wed, 13 Dec 2023 00:00:00 +0000

Editor’s Note

Welcome to the inaugural issue of my newsletter!

Launching this newsletter has been a long time coming. While my blog has been the primary platform for sharing my thoughts and polished work, I’ve held off on creating a newsletter until now. In this space, I aim to offer you a closer look behind the scenes, sharing the journey of how certain projects and transitions came to life.

For instance, the process of building the interactive jq guide was both exhilarating and challenging.

One of the hurdles involved creating the jq evaluator, a puzzle solved by leveraging Aioli.js’s wasm jq distribution and innovative approaches with alpine + web components.

Additionally, migrating the website to Hugo and Cloudflare Pages wasn’t without its complexities, particularly in creating custom includes.

I’ll also be sharing a lighthearted anecdote about my accidental use of jq, where I found myself instinctively generating <option> elements using string interpolation with raw input from ls. Sometimes, a touch of unintended creativity leads to surprisingly usable solutions!

I hope you enjoy this inaugural dive into the less polished yet insightful facets of my work. And as always, I appreciate your ongoing support and feedback.

Warm regards, Ishan

Experiments and Anecdotes

“To a man with jq, everything looks like JSON”

In the world of programming, sometimes our tools become an extension of our thought process. Recently, I found myself unintentionally proving this notion true. While working on a quick prototyping task, I instinctively utilized jq to generate <option> elements using string interpolation with raw input from ls.

The result? A copy-pastable snippet for rapid prototyping. Who knew that accidental encounters with jq could lead to unexpectedly handy solutions?

Check out the full post here

Featured Blog Posts

As this is the first issue, I thought why not highlight some of the articles I’m proudest of so far?

“The secret life of .well-known”
Unveiling the often-overlooked significance of the .well-known folder, this post delves into its role and history.
“Programming ‘with the grain’”
A perspective on programming that aligns with the natural flow of data, this post discusses the impact of following this ‘grain.’ Exploring how this approach enhances program efficiency, it sheds light on the underlying beauty of programming aligned with inherent data structures.
“The Ultimate Interactive JQ Guide”
An achievement I’m particularly proud of, this comprehensive guide boasts 25 interactive examples for learning jq. It’s not just a tutorial; it’s an engaging journey into the heart of this powerful tool.

Closing Thoughts

Thank you for reading the inaugral issue, your support means the world to me!

I’d love to hear your thoughts on this issue.

Feel free to hit reply and share your feedback, ideas, or topics you’d like to see covered in the future. Your input will help shape the upcoming editions!

Until next time,

Ishan

To A Man With `jq`, Everything Looks Like JSON

Thu, 07 Dec 2023 00:00:00 +0000

Featured Photo by energepic.com

I’ve had a love-hate relationship with Bootstrap for a long time. While I love how fast it lets me get going, I absolutely hate the cookie cutter vibe of most of the stuff that I end up making with it. Bootswatch is awesome, and I know that I could customize my theme with SASS variables, but that adds a build step, which is additional complexity that I want to avoid when I am making business/CRUD applications.

Back in the Bootstrap 3 days, there used to be a customizer. I’ve found ones around for Bootsrap 4 as well, but nothing that I quite liked, so I decided to built my own.

The Preview Window

When I’m tweaking a theme, I’d like to have a preview of the theme on a page. The official Bootstrap examples are great for this. I downloaded the examples from the official website and stored them in a folder.

Then I used AlpineJS to create a dropdown menu where users could select an example and it would load the file in an iframe.

<!DOCTYPE html>
<html lang="en">
    <!-- ... -->

<body x-data="{example: ''}">
    <!-- In Navbar -->
    <select x-model="example">
        <option value="/examples/cheatsheet">cheatsheet</option>
        <option value="/examples/blog">blog</option>
    </select>

    <!-- In Body -->
    <iframe x-bind:src="example"></iframe>

    <!-- Alpine -->
    <script src="//unpkg.com/alpinejs" defer></script>
</body>
</html>

IT WORKS!
At this point, all I need to do is populate the select and I’ll be good.

Just one snag though - Since this was a rough prototype all in one file, I’d have to hardcode the options in. I thought about using template and x-for, but it felt like way too much effort.

That was the moment I caught myself typing the following command into my shell:

1
2

ls | 
jq -R -r '"<option value=\"/examples/" + . + "/\">" + . + "</option>"'

It gave me exactly what I wanted:

<option value="/examples/album/">album</option>
<option value="/examples/album-rtl/">album-rtl</option>
<option value="/examples/badges/">badges</option>
<option value="/examples/blog/">blog</option>
<option value="/examples/blog-rtl/">blog-rtl</option>
<option value="/examples/breadcrumbs/">breadcrumbs</option>
<!-- ... many more -->

Copy paste it in, and my switcher works!

Oh Dear..

Yes, that’s right – I used jq, in a manner that might raise a few eyebrows. In hindsight, it felt like trying to hammer a nail with a wrench – a bit unconventional, to say the least.

Breaking down this command, ls is listing the files and directories, and the output is piped (|) to jq. The -R and -r flags are employed to input and output raw strings without JSON formatting. Inside the jq command, there’s an attempt to generate HTML <option> elements by string manipulation.

This quirky misuse of jq in a scenario where a more traditional tool like awk would have been a more fitting choice was both amusing and insightful. After I was done guffawing at what I had just done, I reflected on what just happened. It was a humorous reminder of how our coding environment can shape our thought processes.

As the saying goes:

To a man with a hammer (jq), everything looks like a nail (JSON).

Immersion and Learning

I love learning by immersion. It’s the best way for me to learn anything.

When I immerse myself in a new topic, language, or skill, I feel like I’m absorbing it faster and deeper than any other method.

I enjoy the challenge of figuring things out on my own. I also like the feeling of being part of a community that shares my interest and passion.

Learning by immersion is fun, rewarding, and effective for me.

Side-effects of Immersion

I was so obsessed with jq that I didn’t even realize I was using it for everything, like fitting a square peg through a round hole.

It started when I decided to write the ultimate interactive guide for jq, I wanted to cover every aspect of it, from the basics to the advanced features.

I spent hours and hours reading the documentation, experimenting with different filters, and testing various scenarios. The article languished for months because I wanted it to be perfect.

I learned a lot, but I also got hooked. I started to use jq for every task that involved JSON, even when it was not necessary or optimal. I just loved how jq made it easy and fun to work with JSON.

jq is a great tool, but it is not the only tool. I had become too attached to jq, and I had lost sight of the bigger picture. I needed to take a step back and reevaluate my choices. That’s how I ended up using jq because I had been so caught up with immersing myself.

The Struggle To Keep An Open Mind

Sometimes I wonder if I’m too attached to my own way of thinking.

It’s easy to fall into the trap of applying the same solution to every problem, or seeing the world through a narrow lens. But I don’t want to be that kind of person.

I want to keep an open mind and learn from different perspectives. I want to challenge myself and grow as a human being. That’s why I try to expose myself to new ideas and experiences, even if they make me uncomfortable or confused at first.

I think that’s the only way to avoid becoming stagnant and rigid in my mindset.

Awk is Scary

I don’t know about you, but I will confess that I find awk kind of scary.

It’s a powerful tool for text processing, but it has a weird syntax and a lot of quirks.

I’m usually just copy-pasting the one-liners from StackOverflow when I use awk, and I’m not sure what it will do or if it will backfire.

Maybe I just need more practice, but awk is not my favorite thing to use.

But this is where it would have been more appropriate.

1
2

ls | 
awk '{ print "<option value=\"/examples/" $0 "/\">" $0 "</option>" }'

This awk command constructs HTML <option> elements from each line of the ls output, effectively achieving the desired result with a tool more suited for text manipulation.

Closing Thoughts

So, that’s how I ended up using jq to generate HTML options from a list of files.

It was a silly but fun moment that made me realize how much I rely on this tool for everything JSON-related.

But it also reminded me that sometimes, the best tool for the job is not the one you’re most familiar with, but the one that suits the task at hand.

In tech, we often face problems that require us to think outside the box and use tools in creative or unconventional ways. That’s why it’s important to have a flexible and curious mindset that allows us to learn new skills and explore different solutions.

I hope you enjoyed reading this post. Do you have any stories of using tools in unexpected ways or finding solutions in surprising places? Tell me all about it!

Accidentally deleted your .bashrc?

Tue, 28 Nov 2023 00:00:00 +0000

When beginning to use the terminal, we might accidentally delete our .bashrc file.

What is the .bashrc file?

The .bashrc file is a configuration file used by the bash shell. It contains all the environment variables, user defined functions, aliases and other config including the prompt layout.

It is a full-fledged shell script, so we can have custom commands in it as well, which runs all the commands when a new process of the shell is started.

What happens if I delete it by accident?

It’s totally possible to delete your .bashrc file by accident. If you do this, don’t panic. Your system isn’t bricked.

Bash will default back to the /etc/bash.bashrc file.

This may affect some features (such as colors, prompt, completion, etc), but the shell will still work.

To see what it is like, we can call a custom session of bash with the command:

`1`	`bash --rcfile /etc/bash.bashrc`

Deleting the /etc/bash.bashrc file will make bash use the default settings from the source code. To get a simulation of that, we can run

`1`	`bash --norc`

Restoring Your Config

There’s a special folder in *nixes called /etc/skel.

It contains all the files needed for creating a new user, and thankfully, it contains a sample “skeleton” .bashrc file that we can copy into our home directory.

`1`	`cp /etc/skel/.bashrc ~/.bashrc`

While there isn’t any way to get back our custom commands, we can still get a lot of stuff out as long as you still have a logged in session

The declare built-in command in Bash is used to declare variables and give them attributes. When used with the -x option, it declares the variable as an exported variable, making it available to child processes. Exported variables are passed to any commands or scripts invoked from the current shell. However, if we run declare -x by itself without giving any other parameters, it conveniently prints a list of the exported environment variables for us. The -f flag works in a similiar way, but with functions

# Recover Environment
echo "$(declare -x)" >> ~/.bashrc

# Recover Functions
echo "$(declare -f)" >> ~/.bashrc

# Recover aliases
echo "$(alias)" >> ~/.bashrc

The env command also gives us the same output, so it can be used for this purpose as well.

Making and Restoring Backups

It may be worth taking a backup of your bashrc

1
2
3

cp ~/.bashrc ~/.bashrc.bkp
# Restore
cp ~/.bashrc.bkp ~/.bashrc

A note on Dotfile Management with Stow

Stow is a symlink manager that creates symbolic links from a source directory to a target directory, avoiding conflicts and duplicates To use git and stow for dotfile management, one possible workflow is outlined here.

Closing Thoughts

It’s okay to make mistakes. A fear of messing things up prevents us from playing around to learn. The beauty of learning is in making mistakes. I’ve deleted my .bashrc file before– it’s nothing to be ashamed of!

Have you deleted your .bashrc or other important dotfiles before? Do you already have a workflow with Stow? I’d love to hear about it!.

The Ultimate Interactive JQ Guide

Mon, 06 Nov 2023 00:00:00 +0000

Cover Photo by Pixabay

Has this ever happened to you?

You’ve just received a massive JSON file that looks like it was designed to confuse you. Or maybe you entered a command, and you got so much JSON that it looks incomprehensible.

Important: Level up your jq skills with this accompanying FREE practical project on DataWars — enhance your skills through real-world challenges!

I’ve partnered with DataWars to include Hands On Practical Projects that complement this jq guide, designed to deepen your understanding and enhance your skills in a real-world setting.

The first project, Pokemon Data Analysis is available now.

Click Here to Start Practicing for FREE and LEVEL UP your jq skills

The data you need is buried inside, and you’re dreading the hours it’ll take to extract and clean it up.

I’ve been there. I’ve grepped my way through JSON and written ad-hoc Python scripts to process it for me.

But things don’t have to be like this.

Introduction

jq is one of the best-kept secrets in the data processing world.

Here are some scenarios where jq could swoop in to save your day (and saves mine regularly):

Integrating with APIs in shell scripts often means handling JSON responses, requiring data extraction and manipulation.
Data from different sources may need to be converted to or from JSON format for compatibility.
Managing software configuration files in JSON format can be a regular task.
Extracting data from websites often results in dealing with JSON data that requires parsing and filtering.
Server logs and monitoring data often use JSON, necessitating parsing and analysis.
Infra as Code tools like Ansible and Terraform use JSON-like configurations, requiring management. JSON is a subset of YAML, so every valid JSON file is also a valid YAML file.

All examples are ✨fully interactive✨, so I encourage you to play around!
In fact, I’ll be downright heartbroken if you don’t, because I put a lot of effort into it. You can edit both the input JSON data, and the jq program as well.

Let’s dive in! We’ll start off easy, and get slowly deeper into the weeds.

Basic Operations

Selecting values

Everything in jq is a filter. The dot . is used to select the current object or element, and we can put the property name after it to access a key from an object:

Filtering Arrays

The .[] notation is used to iterate over the elements of an array in a JSON document. It allows you to access each element of an array and perform operations on them.

The select() function is used to filter JSON data based on a specified condition or criteria. It is a powerful tool for extracting specific elements from a JSON document that meet certain conditions.

Similiar to shell scripting, jq works on a pipes-and-filters manner. We use the | to send the data from one filter to the next.

Mapping Arrays

We can use the map function to run any operation on every element of the array and return a new array containing the outputs of that operation:

Combining Filters

The pipe operator | can be used to chain as many filters or functions as we want:

Splitting Strings

We can use the split() function to a split a string on a particular separator character.
Note also the usage of .[0] to select the first index from the split array.

Conditional Logic

We can use if to create expressions

Handling Null Values

Null values can often mess up logic in our scripts, so we can filter them all out using map and select

Formatting Output

Sometimes we don’t want JSON output. We want it in a particular string format.
Note the use of the -r flag, it makes the output raw. Without it, it would be displayed with quote marks around it.

Multiple Outputs

Curly braces create a new object, which we can use for multiple outputs:

Dealing with Nested Items

Photo by cottonbro studio

JSON is very commonly used to store nested objects, and we often need to traverse or manipulate such structures. jq gives us all the tools we need to make it easy:

Important: Itching to practice what you’ve learned? Check out this FREE practical project on DataWars

I’ve partnered with DataWars to include Hands On Practical Projects that complement this jq guide, designed to deepen your understanding and enhance your skills in a real-world setting.

Click Here to Start Practicing for FREE right now

Recursive Descent

We can use .. to recursively descend through a tree of an object.

Filtering Nested Arrays

Flattening Nested JSON Objects

Often, we just want all the key-values, and flattening the object may be the most convenient way to go.

This is an example where the operation we want to do is fairly straightforward, but the program looks way too scary.

Let’s try to break it down:

It takes a JSON input and applies the paths function, which returns an array of all possible paths to the values in the JSON object. Each path is itself an array of keys or indices that can be used to access the value. For example, if the input is {"a": {"b": 1, "c": [2, 3]}}, then the paths function will return [[], ["a"], ["a", "b"], ["a", "c"], ["a", "c", 0], ["a", "c", 1]].
The query then assigns this array to a variable $p using the as keyword, which can be used to store intermediate results for later use.
The query then applies the select function, which filters the array based on a condition. The condition is getpath($p) | type != "object", which means that only the paths that lead to values that are not objects are selected. The getpath function takes a path and returns the value at that path in the JSON input. The type function returns the type of the value, such as “string”, “number”, “array”, or “object”. For example, getpath(["a", "b"]) | type will return “number” for the input {"a": {"b": 1, "c": [2, 3]}}.
The query then applies another pipe, which passes the filtered array to the next filter. The next filter is ($p | join(".")) + " = " + (getpath($p) | tostring), which constructs a string for each path and its corresponding value. The join function takes an array and concatenates its elements with a separator, in this case a dot. The tostring function converts any value to a string representation. The + operator concatenates strings. For example, (["a", "b"] | join(".")) + " = " + (getpath(["a", "b"]) | tostring) will return “a.b = 1” for the input {"a": {"b": 1, "c": [2, 3]}}.
The query then outputs the resulting strings, one per line, to the standard output.

Recursive Object Manipulation

We can use the recurse as well, to traverse a tree.

Complex Object Transformation

Walk through object and apply a transformation conditionally

The walk() function provides a convenient way to traverse a nested object and apply some transformation to it.

Statistical Operations

Photo by Leeloo Thefirst

jq is incredibly handy for doing quick and dirty statistical analysis in the field. Here’s most of the common operations related to that

Sorting Arrays

Sorting an array is a basic operation that is useful for many things in statistics.

Extracting Unique Values from an Array

Extracting unique values from an array is another fairly basic operation that we need for many things.

Calculating Averages

Calculating the mean or average of a dataset is a common statistical operation we may often need to do

Grouping and Aggregating

We can group an array of objects by a particular key and get an aggregated value of the other keys fairly easily:

Filtering after Aggregation

Custom Aggregation with reduce

We can also use reduce to perform a single-output aggregation from an array

Calculating Histogram Bins

We may want to calculate a histogram from an array of data.

Other Common Operations

These are some other common operations I frequently find myself doing every day, but I couldn’t think of a better way to categorize them.

Extracting Values Based on Multiple Conditions

We can combine multiple conditions in a select call. The test() function is used to check if the passed string contains one of the substrings or not.

Formatting Unix Timestamps

Various tools emit Unix Timestamps, and we can use the handy strftime function to format it so it’s easier to understand at a glace.

Enumerating by Top Level Key and Value

Closing Thoughts

Whew! That’s been a long article 😅 If you’re still here, then I appreciate you staying till the very end.

I hope you’ve learned something new, and that you’ll be able to quickly identify use cases for jq in your current workflow and apply your learnings there.

Important: Enjoying this article so far? Check out this FREE practical project on DataWars

How Does This Article Work?

I have used web components to create a custom component <jq-view>.
There is some Javascript in this page which renders all the <jq-view>s when the page is loaded.
The WebAssembly build of jq, as well as all the code for calling out to it is provided by BioWasm.
I have used AlpineJs to make the examples interactive. When the button is clicked, it sends an event to a listener, which makes it run jq and then update the output.
Since I am not good at front-end, this was a substantial learning experience for me.

Get In Touch

If you have any suggestions on how this may be improved, errors that I might have made, or you just want to discuss any other topic, please feel free to email me. I always love to hear from you.

Extra Resources

JQ Manual: The official JQ manual. Covers everything, but a bit difficult to digest.
Learn JQ the Hard Way: A good series of blog posts covering introductory theoretical aspects of jq.
JSON Wrangling with jq: Another interactive series on jq that dives into more detail on the theoretical aspects.

Changelog

2023-11-29
- Add a “Reset” button
- Fixed some examples that were incorrect in the non javascript version of this page
- Added some more resources
- Added explanation for Flattening Nested JSON Objects

The Cult of Not Done Manifesto

Tue, 31 Oct 2023 00:00:00 +0000

To all those who ask why I haven’t been putting more posts out, this is for you

Dear Members of the Cult of Not Done,

I present to you a manifesto of not done. This was inspired by Kio Stark and Bre Pettis, and written over five days because it was, obviously, not done.

The Cult of Not Done Manifesto

There are four states of being. Not knowing, action, refinement, and completion.
Accept that everything is a draft. We have to make it better before we can be done.
We live in the editing stage. Nothing is ever done because it’s always being improved.
Pretending you know what you’re doing is almost the same as knowing what you are doing, but not good enough. So spend all your time making things perfect.
Embrace procrastination. If you wait more than a week to get an idea done, give it more time. Your subconscious needs to think more, and it will give you the answer soon.
The point of being done is to finish, and until we don’t, we can’t get other things done.
If you’re never done, you can throw it away.
Laugh at those who settle. It’s boring and keeps you from being perfection.
People without dirty hands are wrong. Doing something makes you right. The more time you spend trying different things and not being done, the more authority you have on it.
Failure and mistakes are natural and to be expected since you’re not done yet.
Destruction is a force of progress. Start over often with increased scope to make your creation perfect.
If you have an idea, never publish it on the internet, then it will be stolen from you.
In Progress is the engine of more.

Dear past me, use the flags

Wed, 26 Jul 2023 00:00:00 +0000

Dear past me,

I know you’ve just launched your first desktop application. It’s a nightmare isn’t it? Well you’re in for a bigger one.

Your boss calls you, and they need a new feature immediately. You bang it out and send out the update.

Oh yeah, that’s right. You didn’t include an update mechanism. Please use one, you’ll save yourself a lot of pain.

You send out the update to users, but soon you find out everyone hasn’t updated yet. The boss insists that people on an older app version shouldn’t be able to use it.

But you didn’t include a forced version based lockout mechanism either. Well, you’ll remember that for next time!

Desperate times call for desperate measures, and you decide to break the server so it returns bad data to old app versions. Never mind that this permanently corrupts the local database and will require future tech support to help them clear their cache. But that’s future you’s problem.

The boss is happy, the app is chugging along. This cycle repeats a couple more times in various ways.

Save yourself the pain next time. Implement a feature flag / version system. Later, it’ll even help you roll out features to users incrementally.

Use the flags, Luke.

Sincerely, Present me

Analgesia, Pain, and the Shape of The Brain

Sat, 15 Jul 2023 00:00:00 +0000

I was returning from Saturday brunch with my friend a month or so ago. We took the metro back from the restaurant, and it was quite crowded. While disembarking from the metro, my leg slipped (or maybe I was pushed, I don’t know), and got lodged in the gap between the train and the platform. In my vigorous attempts to free myself, I twisted one knee and scraped the other.

Thankfully, I reached home safely, but as I lay in bed that night, writhing in pain, I found my mind wandering back to 2020. The summer of the lockdown, I had found myself quite enamoured with a certain book, PiHKAL: A Chemical Love Story. The main title, PiHKAL, is an acronym that stands for “Phenethylamines I Have Known and Loved.”

PiHKAL, and its sequel, TiHKAL are strange, yet fascinating books. Each is divided into two parts: “The Love Story”, and “The Chemistry”. Most books about science and scientists often focus either on the science itself, or on the scientist and their thoughts or moral flaws of character. But PiHKAL is different. It tells the story of not only the science, but also that of the scientist, and how his research shapes his personality, his personal life, and how they shape the research back. It’s a long (2500+ pages with the sequel) read, but it is worth it.

The Thumb and the Knee

In the first chapter of the book, the narrator recounts his experience as a Navy man during the Second World War. On a destroyer in the middle of the Atlantic in the winter, he contracts an infection on his thumb that causes him immense pain. He required surgery, which wasn’t possible at sea, so all that could be done by the Navy medics was to manage his pain:

The man with the needle would interrupt a good rolling poker game to ask me how I was feeling. I would look at my thumb and say, “It’s a little worse,” or “It’s a little better,” and put my arm out for another morphine shot, then immerse myself again in the poker game. I knew the pain was there, and I could report on the intensity of it with accuracy, but it didn’t bother me. I could play poker, I could deal, I could judge the opposition, and I could bet shrewdly, and I came out ahead more than behind. My left thumb was viciously painful, but the pain just didn’t get in my way. It was fascinating to me that one could be hurting, in agony, and that the administering of a little bit of a chemical that came from some poppy flowers somewhere, could make it all quite unimportant.

This is what is meant by central analgesia. The pain is not deadened; it is still there. The site of action is not the thumb but, rather, the brain. The problem is simply no longer of concern.

Morphine is a pretty remarkable drug.

-PiHKAL

I got to thinking about how I had chosen to manage my own pain:

I put Moov (Diclofenac spray) on the knee which had been twisted.

When diclofenac spray is applied to the skin, it is absorbed through the skin and penetrates into the underlying tissues. Once absorbed, diclofenac works by preventing the body from making chemicals called prostaglandins. Prostaglandins are produced at the place of injury, and are responsible for causing pain, swelling, and inflammation. By not letting the body make prostaglandins, diclofenac helps to reduce these symptoms in the affected area.

In other words, the spray stopped my knee from telling my brain it was “hurt”. It also had a host of “counter-irritants” which overloaded the pain receptors with other sensations (a notable burning one) which simply didn’t let the brain feel the full magnitude of the pain. Scientists call this “working on the periphery.”

“Clever”, I thought to myself.

The spray wasn’t magical by any means, but it did provide enough relief for me to be able to fall asleep.

On Pain

Pain is essentially a way for our body to communicate with us, alerting us that something isn’t quite right. Think of it as a warning system, like a fire alarm. When we experience pain, it’s our body’s way of saying, “Hey, pay attention! Something needs your immediate attention here.”

So, how does pain work? Well, it all starts with specialized nerve endings. These are found throughout our body, including our skin, muscles, organs, and even bones. When we touch a hot stove or getting a paper cut, these spring into action. It sends an electrical signal to the brain.

Now, the brain plays a crucial role in processing and interpreting this signal. It evaluates the intensity, location, and quality of the pain. It also takes into account your previous experiences, emotions, and context to give the pain a subjective meaning.

You see, pain is not just a physical sensation. It’s influenced by various factors, including our emotions, thoughts, and memories. That’s why the same injury can be perceived differently by different people. For example, a minor bruise might be extremely painful to one person but relatively tolerable to another.

Interestingly, pain can also serve as a protective mechanism. Keeping with the stove analogy, the pain you feel immediately forces your hand to avoid further hurt.

Growth and the nature of suffering

“Pain is inevitable, but suffering is optional,” is a Buddhist saying that points to a fundamental truth of existence, which is that pain and affliction are an inherent part of life: we contract illnesses, get wounded, lose our loved ones, our possessions, our social status. But despite the hardships we encounter, the degree of suffering we generate still varies per person. The greatest misfortune hardly affects some people, while the slightest inconvenience leads other people into states of deep agony. Thus, could it be that suffering is something we can manage and doesn’t always have to result from pain?

- Why Pain is Inevitable, but Suffering is Optional, Einzelganger

All growth brings with it naturally, some degree of pain. This is because growth requires us to get out of our comfort zone, which causes us psychological pain. But that pain is justifiable, as long as it helps us to achieve the growth that we are aiming for. This pain should not be confused with suffering.

Suffering is pain that exists without purpose. It has no higher goals.

The Doctor will see you now

“Go to the doctor”, my mother said after observing me hobbling around with my bandaged leg, “He will make it better”

The doctor was sympathetic, emphasizing how lucky I had been to avoid further harm. He then prescribed an X-Ray, and paracetamol.

Paracetamol, also known as acetaminophen, is a commonly used medication to relieve pain and reduce fever. Its exact mechanism of action is not fully understood, but it is believed to work through multiple mechanisms, but primarily in the central nervous system (CNS). It is believed to inhibit the synthesis of prostaglandins in the brain, which can help reduce the perception of pain and lower fever.

Additionally, paracetamol may act on other neurotransmitter systems in the brain, such as the endocannabinoid system, which could contribute to its pain relieving and fever relieving effects too.

It might be useful to take a moment to note the purpose of a fever: The body raises the temperature to make it hard for the infector to survive.

So the doctor isn’t actually making anything better in this case. When we take paracetamol for a fever, or to relieve pain, it doesn’t actually deal with the root cause. It only addresses the symptoms, and fakes our brain into believing that all is okay while the body takes its own sweet time to heal itself.

Closing thoughts

What prompted me to write this was the analogous behaviour to the doctors that I have observed in different software teams that I have worked with. Often, when we are presented with a problem: a bug that is being faced, or some latent issue found in the system, we attempt to treat the symptom, rather than find the root cause:

Too many errors? Just silence it. PHP even comes with a handy @ operator
Program keeps crashing? Just make systemd restart it.
Program OOM’s? Just run it on a bigger machine
Database query takes time? Just cache it

Yes, this is bad practice, but bad practice can’t always be avoided due to organizational issues. Yes, doctors have their perfectly valid reasons for treating the symptoms (because the human body is difficult and it takes a lot of time to fully understand root causes)

Programming "with the grain"

Sun, 09 Jul 2023 00:00:00 +0000

When I was a kid, I observed that most sheets of paper have a sort of “direction” inside them. It’s not possible to tell by looking, but there’s a simple experiment to show:

Rip the same sheet of paper left and right, then up and down. One of the two will have a smooth, straight, nice-looking rip, while the other will have a rough, uneven edge.

Later, as a teenager, I learned that this is called the “grain” of the paper, and that this also applies to shaving and woodworking.

I have observed that computers and programs exhibit similiar characteristics which aren’t obvious at first. Any given computer, platform, problem and program has a “grain” too. A hidden, but present underlying “optimal” (may I even call it “natural”) structure of how data flows and the factorization of the various components that would constitute it. Others seem to have observed it too.

A tryst with locality

Memory can be a funny thing, both when it comes to the self, and to computers. Take the following two pieces of C++ code for example:

// First Example
for (int i = 0; i < size; i++)
{
    for (int j = 0; j < size; j++)
    {
        int element = array[i][j];
        // Do something with the thing item
    }
}

// Second Example
for (int j = 0; j < size; j++)
{
    for (int i = 0; i < size; i++)
    {
        int element = array[i][j];
        // Do something with the thing item
    }
}

Running 100 iterations on a 10x10 array filled with 0s, the first example takes 221ns and the second takes 223ns. That’s not even 1%, and maybe it can be chalked up to statistial error. But what happens when we go bigger?

Array Size	Average 1	Average 2	Best 1	Best 2	Worst 1	Worst 2
10x10	221.52ns	223.42ns	205ns	211ns	480ns	402ns
100x100	24646ns	13296ns	8163ns	8266ns	143007ns	32424ns
1000x1000	806953ns	1.33ms	645242ns	951179ns	1.5ms	3.44ms
10000x10000	81.3ms	675.34ms	70.85ms	632.21ms	101.65ms	776.79ms

The first example performs dramatically better! But why? What is it doing different? Let’s take a closer look:

Imagine a 3 x 3 array. It might look 2D to us:

1
2
3

1,2,3
4,5,6
7,8,9

But, 2D arrays are an abstraction. Memory in computers is single dimensional, so the computer stores this as:

`1`	`1,2,3,4,5,6,7,8,9`

This drastic performance difference is because of the concept of Locality of Reference and how C++ stores your data in memory. Since C++ uses row-major ordering, accessing elements in order of rows (as in the first example) goes with the grain.

Since the elements are stored are more likely to be located nearby in memory, the processor can also “cheat” and load a contiguous chunk of memory into the cache, which allows for much faster access. However, when iterating by column (or against the grain), the processor can’t cache the columns from memory, because the columns don’t exist next to each other.

There’s more to the story than performance though.

The terrible tale of pthreads

When someone says “multithreading”, they almost always mean threading with reference to the POSIX threads model. People often complain about how difficult and error prone the usage of threads are.

However, what most people won’t tell you, is that POSIX threads are against the grain of computers. While threads can run concurrently, the underlying architecture does not directly resemble the interrupt-driven behavior of CPUs. Thread scheduling and synchronization are typically handled by the operating system, abstracting away the low-level hardware details.

The Communicating Sequential Processes (CSP) concurrency model, pioneered by Tony Hoare, focuses on communication and synchronization between concurrent processes rather than explicit thread management. In CSP, processes communicate by sending and receiving messages through channels, and synchronization is achieved through the coordination of message passing. This model aligns more closely with the interrupt-based architecture of CPUs. Go is based on this model.

When in Rome

When in Rome, one should always do as Romans do. When programming, this often extends to following the idioms of the platform and language you are working on, as well as the culture and coding guidelines of the team. That’s why, when in Python land, you follow the Zen of Python, and in Go land, you don’t use a full stack framework. This is because going against the grain of the platform makes it more difficult, and you’ll definitely know when you’re doing it.

As yacoset puts it so elegantly,

You cannot think “Fire rearward missile” and then translate it to Russian, you must think in Russian.

Consider the following program in Python that someone from a C background might write:

sum = 0
for i in range(1, 11):
    sum += i

print("Sum: %d" % sum)

This goes against the grain, and is non idiomatic Python.

1
2
3

sum = sum(range(1, 11))

print(f"Sum: {sum}")

The idiomatic Python version utilizes the built-in sum() function with a range to directly calculate the sum of numbers from 1 to 10. It then uses an f-string to format the output and achieves the same functionality in a more concise and Pythonic way.

How about this code in Go?

package main

import (
	"fmt"
	"io"
	"net/http"
	"sync"
)

func main() {
	urls := []string{"https://example.com", "https://google.com", "https://github.com"}

	var wg sync.WaitGroup
	var mu sync.Mutex

	for _, url := range urls {
		wg.Add(1)
		go func(u string) {
			defer wg.Done()

			resp, err := http.Get(u)
			if err != nil {
				mu.Lock()
				fmt.Printf("Error fetching %s: %s\n", u, err.Error())
				mu.Unlock()
				return
			}

			defer resp.Body.Close()

			body, err := io.ReadAll(resp.Body)
			if err != nil {
				mu.Lock()
				fmt.Printf("Error reading response body from %s: %s\n", u, err.Error())
				mu.Unlock()
				return
			}

			mu.Lock()
			fmt.Printf("URL: %s, Length: %d\n", u, len(body))
			mu.Unlock()
		}(url)
	}

	wg.Wait()
}

In this example, we manually manage the “threads” using the sync.WaitGroup. Each URL is processed by a separate thread, and synchronization is achieved using the sync.Mutex to protect shared access to the console output.

While this approach still accomplishes concurrent processing, it deviates from the idiomatic use of goroutines and channels in Go and involves lower-level thread management.

package main

import (
	"fmt"
	"io"
	"net/http"
)

func main() {
	urls := []string{"https://example.com", "https://google.com", "https://github.com"}

	// Create a channel to receive the results
	results := make(chan string)

	// Launch a goroutine for each URL
	for _, url := range urls {
		go fetchURL(url, results)
	}

	// Collect the results
	for i := 0; i < len(urls); i++ {
		result := <-results
		fmt.Println("Fetched:", result)
	}
}

func fetchURL(url string, results chan<- string) {
	resp, err := http.Get(url)
	if err != nil {
		results <- fmt.Sprintf("Error fetching %s: %s", url, err.Error())
		return
	}

	defer resp.Body.Close()

	body, err := io.ReadAll(resp.Body)
	if err != nil {
		results <- fmt.Sprintf("Error reading response body from %s: %s", url, err.Error())
		return
	}

	results <- fmt.Sprintf("URL: %s, Length: %d", url, len(body))
}

Now, we launch a goroutine for each URL, and each goroutine fetches the content of the URL. The results are then sent back through the results channel. Finally, we collect the results and print them.

This approach, (while still not optimal) aligns with the idiomatic way of utilizing goroutines and channels for concurrent tasks in Go and is much more readable and thus maintainable in the future.

Closing thoughts

Further quoting yacoset,

There’s a thousand computer languages because there’s a thousand classes of problems we can solve with software. In the 1980s, after the Macintosh debut, a hundred DOS products were ported to the new mouse-driven platform by clubbing the Alto-inspired UI over the head and brute-forcing the keyboard-driven paradigms of PCs into the Mac’s visual atmosphere. Most of these were rejected by Apple or the market, and if they came back for a second try they came back because somebody flipped open the spiral-bound HIG and read it sincerely.

As programmers, I think we should take pride in our craft, and always try to program with the grain of the platform we are on. Many benefits await!

On "Natural Selection" and "Survival of the fittest"

Wed, 05 Jul 2023 00:00:00 +0000

The term Natural Selection is often misused, with people saying things like:

natural selection at its finest

he’s about to win a Darwin Award

when they see people doing things that are questionable.

Not only is this a huge festering pool of negativity, but it actually totally misses the point of natural selection.

Natural selection is the process by which those animals and plants which are best suited to the conditions in which they live have more young and live longer.

It is not a matter of only the best individuals of a species surviving, but those that are best adapted to their environment.

On “Survival of the Fittest”

Darwin didn’t say “survival of the fittest”.

Evolutionary biologists certainly aren’t happy with the way it is used by most people, and negative connotations it has in popular culture. It also hides and smoothes over the highly complex nature of natural selection, so modern biologists prefer and almost exclusively use the term natural selection instead.

So where does it come from? Why do we use it the way we do?

Herbert Spencer gave us the phrase in his 1864 book, Principles of Biology, published 5 years after Darwin’s On the Origin of Species.

He correlated his conservative ideas about economics and what Darwin had written about evolution.

This survival of the fittest, which I have here sought to express in mechanical terms, is that which Mr. Darwin has called ‘natural selection’, or the preservation of favored races in the struggle for life.
- Principles of Biology

Spencer later uses it to argue for things like “might is right” in his 1884 work The Man Versus the State:

Thus by survival of the fittest, the militant type of society becomes characterized by profound confidence in the governing power, joined with a loyalty causing submission to it in all matters whatever.
- The Man Versus the State

This work went on to inspire an ideology that we know today as “Social Darwinism”, which is ironically named since it is neither social nor Darwinist. It’s a revolting and disdainful ideology, one that belongs, in the words of journalist Robert Wright, “in the dustbin of intellectual history”. I think that’s a suitable fate for it.

Coming back to Spencer.

In committing this mistake - this “naturalistic fallacy” - the idea that morality and ethics can be derived from nature, he ended up providing the ethics equivalent of a blank check to the capitalists.

As the 19th century (and the Industrial Revolution) chugged along, Spencer’s writings ended up being used to justify laissez-faire economics, war, racism and various heinous political doctrines. It justified the power of capitalists because they where the fittest according to themselves.

However, these ideas pre-date and commonly contradict Darwin’s ideas, and indeed their proponents rarely invoked Darwin in support.

This is all wrong of course. For instance, many species aren’t strong on their own, but still thrive because they cooperate. Humans are pretty weak. Our children are a huge burden on the parents before they grow up. We’re not fast, and neither can we outrun most predators. Most people can’t even outrun a chicken.

A single human is unlikely to thrive. But because we have community and care for each other we managed to succeed as species.

Darwin’s Biggest Mistake

Alfred Russel Wallace suggested Darwin to use the words “survival of the fittest” as an alternative to “natural selection”. He caved.

In 1868, Darwin used the phrase in The Variation of Animals and Plants under Domestication., going so far as to even call it a “bad term”

This preservation, during the battle for life, of varieties which possess any advantage in structure, constitution, or instinct, I have called Natural Selection; and Mr. Herbert Spencer has well expressed the same idea by the Survival of the Fittest. The term ’natural selection’ is in some respects a bad one, as it seems to imply conscious choice; but this will be disregarded after a little familiarity
- The Variation of Animals and Plants under Domestication

In the 1869 (5^th Edition) of On The Origin of Species, Darwin changed it from “Natural Selection” to “Natural Selection, or the Survival of the Fittest”

Closing Thoughts

Stephen Jay Gould (in Life is Beautiful) points out that the survival of a species at a certain time only reflects its fitness at that point in time, and not an absolute value of fitness, which doesn’t exist: there are plenty of genes (or even species) that disappeared for a reason or another, but would be a better fit for today’s conditions than the ones who survived, if they were still here.

Therefore seeing evolution as a continuous improvement of a gene pool towards evergrowing fitness is a misconception anyway. This is why eugenics and the people who bandy it about miss the point, or the randomness of it all. As well as the loooooong timelines and odd (but incredibly egoistic) idea that the most intelligent thing life has produced is somehow stupid compared to imaginary things us stupid things created in our minds.

Addendum: What's a Webfinger?

Tue, 04 Jul 2023 00:00:00 +0000

Following an interesting discussion on programming.dev after the last article, I thought it would be good to add some extra notes regarding Webfinger.

Webfinger provides a standard API for discovering the user profile details and avatar from the username, no matter the software running on the node. The standard Webfinger endpoint is /.well-known/webfinger. It must always be queried with at least the resource.

Some examples using my account on Lemmy (programming.dev) and Mastodon (hachyderm.io):

`1`	`$ curl https://programming.dev/.well-known/webfinger\?resource\=acct:ishanpage@programming.dev \| jq`

{
  "subject": "acct:ishanpage@programming.dev",
  "links": [
    {
      "rel": "http://webfinger.net/rel/profile-page",
      "type": "text/html",
      "href": "https://programming.dev/u/ishanpage"
    },
    {
      "rel": "self",
      "type": "application/activity+json",
      "href": "https://programming.dev/u/ishanpage",
      "properties": {
        "https://www.w3.org/ns/activitystreams#type": "Person"
      }
    }
  ]
}

`1`	`curl https://hachyderm.io/.well-known/webfinger\?resource\=acct:ishands@hachyderm.io \| jq`

{
  "subject": "acct:ishands@hachyderm.io",
  "aliases": [
    "https://hachyderm.io/@ishands",
    "https://hachyderm.io/users/ishands"
  ],
  "links": [
    {
      "rel": "http://webfinger.net/rel/profile-page",
      "type": "text/html",
      "href": "https://hachyderm.io/@ishands"
    },
    {
      "rel": "self",
      "type": "application/activity+json",
      "href": "https://hachyderm.io/users/ishands"
    },
    {
      "rel": "http://ostatus.org/schema/1.0/subscribe",
      "template": "https://hachyderm.io/authorize_interaction?uri={uri}"
    }
  ]
}

While I’m not sure exactly how this is useful, my assumption is that it’s useful during federation, and for generic activitypub clients because different Fediverse software maps usernames and profiles differently.

The secret life of .well-known

Sun, 02 Jul 2023 00:00:00 +0000

My first encounter with the .well-known directory was when Let’s Encrypt first hit the scene and I was learning about VPS-es and how Linux on the server worked.

I noticed that it used certain URLs like .well-known/acme-challenge/3d2f4b8c9d0a4f8b9e6f7d8c9e0a4f8b. This led me down a nice rabbit hole about the ACME protocol, but that’s a story for another time.

Years later, when configuring servers to use paid HTTPs certificates, I noticed the use of the .well-known/pki-validation/ directory for verifying certificates. I filed this knowledge away in my brain as “the .well-known directory is for SSL related stuff” and continued on with my life.

Somewhere in the middle, I noticed that a lot of brute force attacks on servers I monitored would be directed to .well-known URLs. At that point, I had learned about DNS based verification, and quickly moved on from HTTP challenges, and blocked access to the directory in my web server configs. I made this a part of my base setup on every server, and didn’t really think too much about the directory any more.

That was, until very recently.

The Fiasco with NodeInfo

As a recent migrant to the Fediverse after the Reddit API changes shut down my 3rd party app of choice, I joined a fairly interesting Lemmy instance, and I quickly found the distributed nature fascinating. All I could think about was: “I want to see a giant map of all the instances and who is connected to who”, and immediately set out to create a web scraper that could gather this information for me.

Don’t worry, I made sure to respect robots.txt for every site the scraper visited.

Lemmy’s API documentation is fairly poor, and I flailed for quite a while before I discovered the endpoint (/api/v3/site) that would give me the list of federated servers.

That’s when I realized, that Lemmy could and did federate with other Fediverse software like Mastodon, Calckey, Pleroma, WriteFreely, PixelFed and more. And each one of these have different APIs and different endpoints that would give me the data I need.

I needed a way to detect which software each server was running. My first instinct was to use some sort of heuristic, but that’s unreliable, so I continued researching and discovered the NodeInfo protocol.

And wouldn’t you have it, the resource I need to request just happens to be

`1`	`/.well-known/nodeinfo`

At this point, my curiousity was piqued. There was more to this .well-known directory than I had initially thought!

The RFC enters the fray

It turns out, that the .well-known directory is defined in RFC 8615. The intention seems to be to make it like a generic and extendable version of robots.txt

It is increasingly common for Web-based protocols to require the discovery of policy or other information about a host (“site-wide metadata”) before making a request. For example, the Robots Exclusion Protocol http://www.robotstxt.org/ specifies a way for automated processes to obtain permission to access resources; likewise, the Platform for Privacy Preferences [W3C.REC-P3P-20020416] tells user-agents how to discover privacy policy beforehand.

While there are several ways to access per-resource metadata (e.g., HTTP headers, WebDAV’s PROPFIND [RFC4918]), the perceived overhead (either in terms of client-perceived latency and/or deployment difficulties) associated with them often precludes their use in these scenarios.

When this happens, it is common to designate a “well-known location” for such data, so that it can be easily located. However, this approach has the drawback of risking collisions, both with other such designated “well-known locations” and with pre-existing resources.

To address this, this memo defines a path prefix in HTTP(S) URIs for these “well-known locations”, /.well-known/. Future specifications that need to define a resource for such site-wide metadata can register their use to avoid collisions and minimise impingement upon sites’ URI space.

There’s a whole bunch of stuff that you can do with stuff in the .well-known folder, like:

and of course, let’s not forget about Webfinger.

“A webfinger? Is that what I use to poke someone on Facebook?” - Anonymous

UPDATE 2023-07-04: I have published a follow-up - Addendum: Webfinger

UPDATE 2023-07-07: This article was featured in tldr.tech, and as of time of writing I have had over 6500 hits on this post. I am overwhelmed and moved by this, and by all the positive comments I have received. I never imagined that my little blog would ever be read by so many people.

Get frequency distribution of countries in a list of newline separated IP addresses in Linux

Sat, 01 Jul 2023 00:00:00 +0000

Assuming that the IP addresses are in a file called list.txt in a format such as below:

104.28.29.66
92.40.175.51
116.255.32.201
91.150.51.170
108.167.20.103
5.187.159.175
166.198.250.121
74.78.184.213
...

A prerequisite for this is the geoiplookup command, which is available in the geoip-bin package on most distros.

Run the following command to get the frequency distribution:

`1`	`cat list.txt \| xargs -n 1 geoiplookup \| sed 's/GeoIP Country Edition: //g' \| sort \| uniq -c \| sort -r`

This will give you an output as:

    462 US, United States
    124 DE, Germany
     62 CA, Canada
     47 GB, United Kingdom
     34 AU, Australia
     31 NL, Netherlands
     23 FR, France
     19 IN, India
     18 SE, Sweden
     14 FI, Finland
     10 BE, Belgium
      9 NO, Norway
      9 IT, Italy
      9 BR, Brazil
      7 SG, Singapore
      7 PT, Portugal
      7 IL, Israel
      7 CH, Switzerland
      7 AT, Austria
      6 IR, Iran, Islamic Republic of

Get Docker Containers running in a particular network

Sat, 24 Jun 2023 00:00:00 +0000

The following script can be used to start list the running containers inside a particular network along with their ips.

1
2
3

#! /usr/bin/zsh
echo "Image\tHostname\tIP";
for N in $(docker ps -q) ; do echo "$(docker inspect -f '{{.Config.Image}}\t{{ .Config.Hostname }}\t' ${N}) $(docker inspect -f '{{range $i, $value := .NetworkSettings.Networks}}{{if eq $i "custom_network_name"}}{{.IPAddress}}{{end}}{{end}}' ${N})"; done

Usage example:

`1`	`./network-hosts`

Bonus tip:
Pipe the output into column -t like so to get a pretty formatted tabular output.

`1`	`./network-hosts \| column -t`

Parsing single level yaml files into a Lua table

Fri, 23 Jun 2023 00:00:00 +0000

Given a yaml-like file containing key-value pairs like this:

1
2
3

key1: value1
key2: value2
key3: value3

The following function will load it and return the data as a Lua table.

function read_yaml_file(filename)
  local file = io.open(filename, "r")
  if not file then
    return nil, "Failed to open file: " .. filename
  end

  local data = {}
  for line in file:lines() do
    local key, value = line:match("(%w+):%s*(.+)")
    if key and value then
      data[key] = value
    end
  end

  file:close()
  return data
end

Start a container in a particular docker network

Thu, 22 Jun 2023 00:00:00 +0000

The following script can be used to start a container inside a particular network.

Usage example:

1
2

./start-in-network redis
./start-in-network vad1mo/hello-world-rest 5050

#!/usr/bin/zsh

if [ $# -eq 0 ]
then
    echo "syntax: ./start-in-network container-name [container port binding]"
    echo "You must provide at least container name"
    exit 1
fi

docker_container_id=""
if [ $# -eq 1 ]
then
    docker_container_id=$(docker run -d $1)
else
    docker_container_id=$(docker run -d -p $2:$2 $1)
fi

docker network connect custom_network_name docker_container_id

Using different Github accounts with different private keys on Linux

Wed, 21 Jun 2023 00:00:00 +0000

Create a file ~/.ssh/config

If you have 2 keys, for example: id_rsa for your personal, and id_work for your work, set the config as:

Host github-work
    HostName github.com
    IdentityFile ~/.ssh/id_work
    IdentitiesOnly yes

Now, when cloning or adding remote, change the github.com in the clone url is changed to github-work

`1`	`git clone git@github-work:username/whatever.git`

Get leads collected by a particular lead ad form between two timestamps using Facebook Graph API

Tue, 20 Jun 2023 00:00:00 +0000

Given two unix timestamps, this Python script fetches all the leads received by a particular Facebook Lead Ads form using the Facebook v14 Graph API.

import json
import requests

# Constants
ACCESS_TOKEN="YOUR_ACCESS_TOKEN"
LIMIT=500
FROM_TIMESTAMP=1678613400
UPTO_TIMESTAMP=1678704000
FORM_ID=12341241244

def create_api_url(form_id):
    return f"https://graph.facebook.com/v14.0/{form_id}/leads"

def filter_tpl(gt, lt):
    f = [
        {
            "field": "time_created",
            "operator": "GREATER_THAN",
            "value": gt
        },
        {
            "field": "time_created",
            "operator": "LESS_THAN",
            "value": lt
        }
    ]
    return json.dumps(f)

def do_query(form_id, gt, lt):
    url = create_api_url(form_id)

    querystring = {"access_token":ACCESS_TOKEN,"limit":str(LIMIT),
        "filtering":filter_tpl(gt, lt)}

    payload = ""
    response = requests.request("GET", url, data=payload, params=querystring)

    print(response.text)

if __name__ == '__main__':
    do_query(FORM_ID, FROM_TIMESTAMP, UPTO_TIMESTAMP)

Hello World

Tue, 20 Jun 2023 00:00:00 +0000

Many people who know me ask me why I haven’t got a personal website or blog. I often don’t know what to tell them. It’s certainly not for a lack of trying.

Here’s one of the very first blogs I started to help my 7th grade classmates learn programming which they weren’t able to understand in school:

I fumbled around various other blogs, but never managed to be able to post consistently. I was either too lazy, having other things going on in life, or just plain scared that people would judge me. (this also explains why my github is so sparse, but I plan to change that soon)

I settled for having a sweet and simple landing page, based on a theme from HTML5Up:

While decidedly amateurish and minimal, it’s also woefully out of date (I last used VueJS in 2019). So, it’s time for a refresh.

Goals and wishes

Rescue old blog posts from whatever previous blogs I have in my backups and post them here
A “Projects” section where I can post my projects and project related writeups
Have fun indieweb and fediverse things
Add a proper RSS feed and webmentions support
A “Grimoire” where I can post code snippets and quick explainers for my future self
Semi-regular blog posting frequency (fingers crossed!)

Is this an ambitious list? Probably. But that’s okay. Let’s see how it goes. I’m excited to see what the future holds for me.

Fix WSL time drift

Wed, 08 Mar 2023 00:00:00 +0000

There is a bug in WSL2 that causes the clock inside the VM to drift behind the actual clock on the machine.

This can lead to many unexpected issues, ranging from harmless (git commit messages having old timestamps) to severe (SSL connection failures).

We can sync the VM clock using the command

`1`	`sudo hwclock -s`

We can also add the command as a cron job every 5-10 minutes to make sure that the clock doesn’t drift too far back.

About

Sun, 06 Mar 2022 00:00:00 +0000

Hi There, I’m Ishan.

I was fascinated by coding from a young age. I taught myself how to code from the ground up, and I have never stopped learning and exploring new possibilities.

Over the past ten years, I have worked on various projects that challenged me to grow as a coder and a problem-solver. I have learned how to build software that is reliable, scalable and performant, and that serves the needs of millions of users.

Thank you for visiting my blog, where I share my journey, thoughts, and experiences with you. I hope you find something useful and inspiring here.

Currently, I am working on DDoS mitigation systems at Asians Group LLC. I have previously worked on systems and websites using OpenCV, Python, Django, PHP, Laravel, Vue, Eleventy, and many others.

I’m also finally working towards a Bachelor’s Degree in Data Science and Applications from the Indian Institute of Technology, Madras.

The beautiful photo was taken by my friend Sabyasachi Das in scenic Pondicherry, India.

Featured In

About This Website

This website is the latest iteration of many.

FAQs

How do you deal with bugs that just won’t budge? Do you bribe them?

Yep, you caught me. I bribe them. I’m dirty.

Do you have a secret coding dance that boosts productivity?

It’s more of a secret keyboard rhythm that syncs perfectly with the code’s heartbeat. Sadly, no dance moves… yet.

How do you manage to stay sane during those long debugging sessions?

Insanity is relative! But seriously, music, memes, and the occasional staring contest with the screen keep me going.

You can subscribe to updates via RSS or you could subscribe to the newsletter instead.

What did you use to make this website?

This website is made with Hugo, using the excellent Stack theme with my own customizations.

What other blogs do you like?

Too many to list! Maybe one day I will try to collate them.

What is the best programming language?

Haskell, obviously.

Is it true you’ve had staring contests with your code? Who won?

It’s a tie between me and the code; we both blinked and then had to debug why.

Do you believe in horoscopes?

Absolutely! My horoscope for today: “You will encounter a syntax error, but fear not, for the solution is just a semicolon away!”

Why do you hate SPAs?

I don’t necessarily hate SPAs - I just believe there is a right and a wrong time to SPA.

Why do you hate Javascript?

Ecosystem catch-up overload. I don’t hate Javascript - but it’s hard to keep up with all the happenings.

Elsewhere

Articles

Sun, 06 Mar 2022 00:00:00 +0000

Discord Notifications from Systemd

Fri, 15 Oct 2021 00:00:00 +0000

Add the following lines in your service definition file (/etc/systemd/system/service_xyz.service) in the [Service] section:

1
2

ExecStartPost=/home/service_xyz/hooks.sh start
ExecStopPost=/home/service_xyz/hooks.sh stop

In /home/service_xyz/hooks.sh, put

#!/bin/bash

hook_url=https://discord.com/api/webhooks/xxxxxxxxxxxx/yyyyyyyyyy

case "$1" in
 start)
  curl -H "Content-type: application/json" \
  -X POST -d \
  '{
    "content":"Hook: <NAME>; Action Start"
  }' $hook_url
   ;;
 stop)
  curl -H "Content-type: application/json" \
  -X POST -d \
  '{
    "content":"Hook: <NAME>; Action Stop"
  }' $hook_url
   ;;
esac

exit 0

Generate PEM file for any Linux user

Sat, 18 Sep 2021 00:00:00 +0000

Quickly and easily generate a key pair for passwordless connection. Also works perfect as git deploy keys!

Generate the keys

DO NOT DO THIS AS ROOT

Run the command to generate the keys

`1`	`$ ssh-keygen -P "" -t rsa -b 4096 -m pem -f my-key-pair`

-b flag sets the bits, 4096 is recommended
-m pem is needed to generate a file in RSA Private Key format, not in OpenSSH Private Key format
-f specifies the output key pair
replace my-key-pair with the name of your key pair (preferably deploy_key or access_key)

This will generate two files:

my-key-pair - this is your PEM file, copy paste the contents
my-key-pair.pub - this is the corresponding public key file

Note: You may want to rename the my-key-pair to my-key-pair.pem for convenience.

In order to be able to log in using the generated PEM file, you should add the public key to the authorized_keys list. To do that, run

# Ignore this step if you have other public keys added
$ touch ~/.ssh/authorized_keys

$ cat my-key-pair.pem >> ~/.ssh/authorized_keys

Build and Deploy Eleventy via SFTP using Github Actions

Wed, 15 Sep 2021 00:00:00 +0000

Navigate to Repository > Settings > Secrets > New Repository Secret
Set the name as SSH_PRIVATE_KEY
Paste the contents of the key file

# .github/workflows/build.yaml

name: Build and Deploy PROJECT_NAME

on:
  push:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest

    strategy:
      matrix:
        node-version: [14.x]
    
    steps:
      - uses: actions/checkout@v2

      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v1
        with:
          node-version: ${{ matrix.node-version }}
      
      - name: Install dependencies & build
        run: |
          npm ci
          npm run build:assets
          npm run build:site          
      
      - name: Deploy build
        uses: wlixcc/SFTP-Deploy-Action@v1.2.1
        with:
          username: 'domain.user'
          server: 'server.host'
          ssh_private_key: ${{ secrets.SSH_PRIVATE_KEY }}
          local_path: './dist/*'
          remote_path: 'site_home_dir'
          args: '-o ConnectTimeout=5'

Install Postgres into XAMPP on Windows

Wed, 08 May 2019 00:00:00 +0000

Setting up a local Postgres development environment

1. Install Postgres

Download the installer from EnterpriseDB
Run the installer
Install Postgres into C:\xampp\pgsql{version_number} folder

2. Enable PostGres modules for PHP

Open php.ini file located in C:\xampp\php.

Uncomment the following lines in php.ini

1
2

extension=php_pdo_pgsql.dll
extension=php_pgsql.dll

3. Install Adminer

Download Adminer
Place the PHP file into HTDOCS

4. Wrapping up

Restart Apache
Adminer is now accessible here

3 ways to render inline lists in PHP

Mon, 15 Apr 2019 00:00:00 +0000

Given a list of items and a separator, render it in a single line, with separators between.

$items = [ 
    "apples",
    "oranges",
    "bananas"
];

$separator = " / ";

Method 1: Basic `foreach`

This approach is the one that comes to mind first, but it is incorrect because it will emit an additional separator after the last element.

1
2
3

foreach ($items as $value) { 
    echo $value . $separator;
}

Output:

`1`	`apples / oranges / bananas /`

Method 2: Using `implode()`

Cleaner and shorter, this usage is appropriate for most applications:

`1`	`echo implode($separator, $items);`

Output:

`1`	`apples / oranges / bananas`

While the implode() method is great for most purposes, it does break down when there is heavy templating involved.

Method 3: Using `foreach` and

When heavier templating functionality is required, it’s best to use

foreach ($items as $key => $value) {
    echo $value;

    if ($key !== array_key_last($items)) {
        echo $separator;
    }
}

The third method has the same output as the second one, but gives you more flexibility in how you might organize your templates.

How to install and optimize Apache with mpm_event, php5-fpm and MariaDB

Tue, 03 Nov 2015 00:00:00 +0000

A note to the reader

This post is a legacy post. The legacy posts that are available on this website were written many years ago. These posts are made available here for archival purposes only.
They reflect the age I was, and the level of knowledge that I had when I wrote them, and they may contain outdated information, so please keep that in mind as you proceed to read this article.

Original Title: How to install a highly optimized LAMP stack with mpm_event and MariaDB

Apache is today more important as a web server than ever before. Most PHP scripts assume (or even require) apache to be installed in order to run. However, for most people running a VPS with less than a GB of RAM, having to run apache is a deal breaker. The same applies for the stock MySQL configuration. This tutorial will help you to install Apache with the event MPM + PHP [fastcgi] + MariaDB 5.5

Step 1 - Install Prerequisites

This tutorial requires root. If you’re not already root, run this command:

sudo su

and gain root priviledges. Alternatively, you may prefix the commands with sudo also. We will use the aptitude package manager as a replacement for apt-get. Since it is not available by default, go ahead and install it.

`1`	`apt-get install -y aptitude`

Step 2 - Install Apache

Apache is sometimes included by default by ubuntu, however aptitude will take care of removing the preform mpm when we install the apache fastcgi module, and the event mpm

`1`	`aptitude install -y libapache2-mod-fastcgi apache2-mpm-event`

Then, we disable the build in apache php modules as they consume too much precious memory

1
2
3

a2dismod php4 # PHP v4 is outdated. It should be criminal to even have it on a system
a2dismod php5 # mod_php is not usable with worker mpm, therefore we disable it
a2dismod fcgid # fcgid is not required. We will set up fastcgi later.

Then, we will enable the mod_fastcgi, mod_ssl, mod_actions and mod_rewrite

a2enmod fastcgi
a2enmod ssl
a2enmod actions
a2enmod rewrite

Step 3 - Install PHP

We will be running PHP as a FastCGI process. We will configure FastCGI to run as a separate server and have a 120 second timeout.

First, we configure the apache2 fastcgi module.

`1`	`nano /etc/apache2/mods-available/fastcgi.conf`

<IfModule mod_fastcgi.c>
FastCgiIpcDir /var/lib/apache2/fastcgi
# Set FPM to run externally
FastCGIExternalServer /srv/www/fcgi-bin.d/php5-fpm -pass-header Authorization -idle-timeout 120 -socket /var/run/php5-fpm-www-data.sock
Alias /php5-fcgi /srv/www/fcgi-bin.d
AddType application/x-httpd-php5 .php
# FPM will handle all PHP files
<FilesMatch "\.php$">
SetHandler php-fpm
</FilesMatch>
Action php-fpm /php5-fcgi/php5-fpm

# Prevent external access to FPM directory
<Location "/php5-fcgi/php5-fpm">
    Order deny,allow
    Deny from All
    Allow from env=REDIRECT_STATUS
</Location>

</IfModule>

Finish off by creating the FPM directory

`1`	`mkdir -p /srv/www/fcgi-bin.d`

Then, we install the php packages Base PHP packages. Essential to run PHP

`1`	`aptitude install -y php5-fpm php5-common php-apc php5-mysqlnd php5-dev`

Secondary, but useful PHP modules/packages. Feel free to omit

`1`	`aptitude install -y php5-memcache php5-curl php5-mcrypt php5-xsl php5-gd php5-imagick php5-snmp php5-xmlrpc`

And, we’re done installing PHP

Step 4 - Install MariaDB

We will be using MariaDB as our database server. It is an enhanced, drop-in replacement for MySQL that is developed by the original team and is more performant

Install MariaDB

`1`	`aptitude -y install mariadb-server mariadb-client`

Secure MariaDB

`1`	`mysql_secure_installation`

And we’re done!

Step 5 - Pre-Optimization

Before we optimize, we need to restart apache2 and php.

1
2

service php5-fpm restart
apache2ctl graceful

Step 6 - Optimize Apache

The only optimization apache really needs is the config optimization. Edit the config file:

`1`	`nano /etc/apache2/apache2.conf`

...
LockFile ${APACHE_LOCK_DIR}/accept.lock
...
PidFile ${APACHE_PID_FILE}
...
Timeout 300
...
KeepAlive On
...
MaxKeepAliveRequests 100
...
KeepAliveTimeout 5
...
<IfModule mpm_worker_module>
    StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadLimit          64
    ThreadsPerChild      25
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>
...
<IfModule mpm_event_module>
    StartServers          1
    MinSpareThreads       2
    MaxSpareThreads       5 
    ThreadLimit           20
    ThreadsPerChild       20
    MaxClients            60
    MaxRequestsPerChild   5000
</IfModule>
...
# These need to be set in /etc/apache2/envvars
User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}

...

AccessFileName .htaccess

...
#
# The following lines prevent .htaccess and .htpasswd files from being 
# viewed by Web clients. 
#
<Files ~ "^\.ht">
    Order allow,deny
    Deny from all
    Satisfy all
</Files>

...
<Directory />
    Options -Indexes -FollowSymLinks
    AllowOverride All
    Order allow,deny
    allow from all
</Directory>
...
DefaultType None
...
HostnameLookups Off

...
ErrorLog ${APACHE_LOG_DIR}/error.log
...
LogLevel warn
...
# Include module configuration:
Include mods-enabled/*.load
Include mods-enabled/*.conf
...
# Include list of ports to listen on and which to use for name based vhosts
Include ports.conf
...
#
# The following directives define some format nicknames for use with
# a CustomLog directive (see below).
# If you are behind a reverse proxy, you might want to change %h into %{X-Forwarded-For}i
#
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
...
# Include generic snippets of statements
Include conf.d/
...
# Include the virtual host configurations:
Include sites-enabled/

Step 7 - Optimize FPM

Note: Most of these steps are done with automated linux tools like sed which make it easy to edit the configuration files. If you want to know what’s going on behind the scenes, refer to the comments above each command block.

Before we optimize FPM, we need to stop the service

`1`	`service php5-fpm stop`

The FPM configuration file is lengthy and complex. To quickly do the configuration automatically, run:

php_fpm_conf="/etc/php5/fpm/pool.d/www.conf"
# Configure these values to your liking. Optimized for 512MB RAM VPS
FPM_MAX_CHILDREN=5
FPM_START_SERVERS=1
FPM_MIN_SPARE_SERVERS=1
FPM_MAX_SPARE_SERVERS=2
FPM_MAX_REQUESTS=5000

pm.max_children is the configuration value that controls the maximum number of server processes that will be started by FastCGI. 5 is usually enough, but it might be a good idea to set it to 10 or 15 for a high traffic site on a high-end vps

`1`	`sed -i 's/^pm.max_children.*/pm.max_children = '${FPM_MAX_CHILDREN}'/' $php_fpm_conf`

pm.start_servers is the number of servers to be initially started by FastCGI. 1 is nearly always enough. Try 2 or 3 for higher traffic sites. Make sure that is it less that pm.max_children.

`1`	`sed -i 's/^pm.start_servers.*/pm.start_servers = '${FPM_START_SERVERS}'/' $php_fpm_conf`

pm.min_spare_servers is the minimum number of server processes FastCGI will have ready to handle additional load. Each server process is respawned when it hits pm.max_requests. It is here that this value becomes useful. It ensures that the site will not go offline for even a small amount of time, and will remain performant even under load. 1 is enough for small sites (< 5000 views / minute). For larger sites, try 5 or 6.

`1`	`sed -i 's/^pm.min_spare_servers.*/pm.min_spare_servers = '${FPM_MIN_SPARE_SERVERS}'/' $php_fpm_conf`

pm.max_spare_servers exists for only one reason: to prevent FastCGI from consuming too many resources in the event of high load. This is the number of maximum server processes that will be kept ready by FastCGI. Make sure it is at least half of pm.max_children and greater than pm.min_spare_servers. 2 is OK for a small site. Try 9 or 10 for a large site

`1`	`sed -i 's/^pm.max_spare_servers.*/pm.max_spare_servers = '${FPM_MAX_SPARE_SERVERS}'/' $php_fpm_conf`

pm.max_requests controls the maximum number of hits a FastCGI process is allowed to serve before it is killed and respawned. This value is there in place to deal with the fact that over time, these processes leak memory. Periodically killing and respawning them allows us to keep the site snappy. 5000 is a pretty good value to begin with, give or take a couple hundreds. If pm.min_spare_servers is < 5, and this value is < 5000, the site will become more sluggish. Never set it to a 3 digit or less number. Always ensure that it is >= 5000

`1`	`sed -i 's/\;pm.max_requests.*/pm.max_requests = '${FPM_MAX_REQUESTS}'/' $php_fpm_conf`

Change to socket connection for better performance

`1`	`sed -i 's/^listen =.*/listen = \/var\/run\/php5-fpm-www-data.sock/' $php_fpm_co`

Now, configure the php.ini file.

php_ini_dir="/etc/php5/fpm/php.ini"
# Configure these values to your liking. Optimized for 512MB RAM VPS
PHP_MEMORY_LIMIT=192M
PHP_MAX_EXECUTION_TIME=120
PHP_MAX_INPUT_TIME=300
PHP_POST_MAX_SIZE=50M
PHP_UPLOAD_MAX_FILESIZE=50

max_execution_time is the maximum amount of time (in seconds) that a script is allowed to execute for, before it is killed. 120 is a big enough number to do most heavy calculations, but this number may be increased if many computations are required to be performed or heavy downloading needs to be done.

`1`	`sed -i 's/^max_execution_time.*/max_execution_time = '${PHP_MAX_EXECUTION_TIME}'/' $php_ini_dir`

memory_limit is the maximum amount of memory (in MB) that a PHP script is allowed to take up. 192 is more than enough for most jobs, but you may consider raising it if you want to do heavy computations.

`1`	`sed -i 's/^memory_limit.*/memory_limit = '${PHP_MEMORY_LIMIT}'/' $php_ini_dir`

This is the maximum amount of time (in seconds) for which a client can upload data to the server through a keep-alive connection. 300 seconds (or 5 minutes) is usually enough. You may want to set it to something like 6300 (2 hours) if you plan on uploading huge files.

`1`	`sed -i 's/^max_input_time.*/max_input_time = '${PHP_MAX_INPUT_TIME}'/' $php_ini_dir`

post_max_size is the maximum size (in MB) of the data that can be sent to the server through a POST request. 50 MB is usually more than enough for most needs, but feel free to change to will.

`1`	`sed -i 's/^post_max_size.*/post_max_size = '${PHP_POST_MAX_SIZE}'/' $php_ini_dir`

upload_max_filesize is the maximum size of an individual file (in MB) that can be uploaded. Change at will.

`1`	`sed -i 's/^upload_max_filesize.*/upload_max_filesize = '${PHP_UPLOAD_MAX_FILESIZE}'/' $php_ini_dir`

Don’t expose PHP. More of a security fix than memory optimization

`1`	`sed -i 's/^expose_php.*/expose_php = Off/' $php_ini_dir`

Disable potentially dangerous PHP functions that may allow attackers to execute arbitary code on your machine.

`1`	`sed -i 's/^disable_functions.*/disable_functions = exec,system,passthru,shell_exec,escapeshellarg,escapeshellcmd,proc_close,proc_open,dl,popen,show_source/' $php_ini_dir`

Now we restart all the services

1
2
3

service php5-fpm start
service php5-fpm restart
apache2ctl graceful

Step 8 - Optimize MariaDB

MariaDB configuration is quite easy to optimize. Just dropping InnoDB can cause a huge performance gain.

`1`	`nano /etc/mysql/my.cnf`

[mysqld]
default-storage-engine = myisam
key_buffer = 1M
query_cache_size = 1M
query_cache_limit = 128k
max_connections=25
thread_cache=1
skip-innodb
query_cache_min_res_unit=0
tmp_table_size = 1M
max_heap_table_size = 1M
table_cache=256
concurrent_insert=2 
max_allowed_packet = 1M
sort_buffer_size = 64K
read_buffer_size = 256K
read_rnd_buffer_size = 256K
net_buffer_length = 2K
thread_stack = 64K
END

Now we restart the service

`1`	`service mysql restart`

Step 9 - Reboot

Reboot the server to get better performance

`1`	`shutdown -r now`

Step 11 - Set up a non-root sudo-enabled user

We will use this user’s home directory to contain our vhosts. Add the User

1
2

adduser demo # Replace demo with your name
# Like `adduser ishan`

Grant him priviledges

`1`	`gpasswd -a demo sudo`

Step 12 - Add a virtualhost

Replace “example.com” with your domain Replace “demo” with your user Create an FPM user

DOMAIN_OWNER=demo
cp /etc/php5/fpm/pool.d/{www.conf,$DOMAIN_OWNER.conf}
# Quickly edit the FPM file
# Create a new FPM pool with same name as user
sed -i 's/^\[www\]$/\['${DOMAIN_OWNER}'\]/' /etc/php5/fpm/pool.d/$DOMAIN_OWNER.conf
# set it to listen on a unix socket
sed -i 's/^listen =.*/listen = \/var\/run\/php5-fpm-'${DOMAIN_OWNER}'.sock/' /etc/php5/fpm/pool.d/$DOMAIN_OWNER.conf
# set permissions
sed -i 's/^user = www-data$/user = '${DOMAIN_OWNER}'/' /etc/php5/fpm/pool.d/$DOMAIN_OWNER.conf
sed -i 's/^group = www-data$/group = '${DOMAIN_OWNER}'/' /etc/php5/fpm/pool.d/$DOMAIN_OWNER.conf
sed -i 's/^;listen.mode =.*/listen.mode = 0660/' /etc/php5/fpm/pool.d/$DOMAIN_OWNER.conf
sed -i 's/^;listen.owner =.*/listen.owner = www-data/' /etc/php5/fpm/pool.d/$DOMAIN_OWNER.conf
sed -i 's/^;listen.group =.*/listen.group = www-data/' /etc/php5/fpm/pool.d/$DOMAIN_OWNER.conf

Restart FPM

`1`	`service php5-fpm restart`

Create a public_html for your domain

`1`	`mkdir -p /home/demo/example_com/public_html`

Create the log files:

`1`	`touch /home/demo/test_digitalocean_tk/logs/{access.log,error.log}`

Set permissions

chown demo:demo /home/demo/
chown -R demo:demo /home/demo/example_com
# Allow execute permissions to group and other so that the webserver can serve files
chmod 711 /home/demo/
chmod 711 /home/demo/example_com

Create the VirtualHost file

`1`	`nano /etc/apache2/sites-available/example_com`

<VirtualHost *:80>

    ServerName example.com
    ServerAlias www.example.com
    ServerAdmin admin@example.com
    DocumentRoot /home/demo/example_com/public_html/
    ErrorLog /home/demo/example_com/logs/error.log
    CustomLog /home/demo/example_com/logs/access.log combined

    FastCGIExternalServer /home/demo/example_com/php5-fpm -pass-header Authorization -idle-timeout 120 -socket /var/run/php5-fpm-demo.sock
    Alias /php5-fcgi /home/demo/example_com

</VirtualHost>

Enable the virtualhost

`1`	`ln -s /etc/apache2/sites-available/example_com /etc/apache2/sites-available/example_com`

Test the virtualhost

`1`	`nano /home/demo/example_com/public_html/info.php`

1
2
3

<?php
phpinfo();
?>

navigate to your domain in the browser and see that it’s working correctly.

Conclusion

Now that you know how to install and optimize a LAMP stack, get cracking and create a great site that will be big enough to crash this setup!

Scraping the MPB website - A Postmortem

Thu, 09 Oct 2014 00:00:00 +0000

A note to the reader

This post is a legacy post. The legacy posts that are available on this website were written many years ago. These posts are made available here for archival purposes only.
They reflect the age I was, and the level of knowledge that I had when I wrote them, and they may contain outdated information, so please keep that in mind as you proceed to read this article.

Introduction

Last week, I scraped the School Website clean looking for report cards. I wanted the data to do some statistics (the website is very spartan) and, much to my pleasure, I was successfully able to get all the data I need.

The Problem

I need the report card data in JSON format. The format given is, quite unsurprisingly, Bad HTML

Problem 1: Bad HTML

The Website is built with ASP.NET. What better can I expect? Tables in tables

  <td><table width='100%' border='0' cellspacing='0' cellpadding='0' style='border-bottom: 1px #000000 solid'><tr><td style='border-right: 1px #000000 solid'>&nbsp;</td><td style='font-family: Arial, Helvetica, sans-serif; font-size: 12px; font-weight: bold; color: #000000; padding: 3px 3px 3px 5px;border-right: 1px #000000 solid'>Subject</td>

That’s an example of the stuff I had to deal with

Problem 2: Inconsistent ID Numbers

Come on. Who would expect people from Class 11 and 12 to have joined the school in 2014?

The Solution

Solution 1: Xpath

XPath is what saved me. True, it took some analysis, but I got a working thing ready

Solution 2: Python + lxml

Python was used to write the bruteforcing script. It’s what helped me to make it easier to work with the tedious job

Lxml was used to parse the Xpath and extract the data

Finally some helper scripts made the data into yearly chunks, classes, segregated and organized it into nice self-contained JSON files

Solution 3: Koding

My internet is too slow to allow the script to be practical. A VPS from Koding helped me to get the job done in minutes instead of weeks.

The entire extraction took 15 Mins on the VPS. There were some >2500 Records, My connection can do 25.1 records in 15 mins

What went Right

The extraction and processing

What went Wrong

A lot of things

Firstly, I could have used threading. It would have made everything much much faster
Secondly, I should have used Scrapy
Thirdly, I should have done my homework about the subjects and classes
Fourthly, I did a lot of useless year-crawling like 2000 and 1998

Downloads

Ok. Here you go: data download link is now removed

Gamedev Ideas

Sun, 07 Sep 2014 00:00:00 +0000

A note to the reader

This post is a legacy post. The legacy posts that are available on this website were written many years ago. These posts are made available here for archival purposes only.
They reflect the age I was, and the level of knowledge that I had when I wrote them, and they may contain outdated information, so please keep that in mind as you proceed to read this article.

Introduction

Hello there. I’ve recently spent quite some time thinking up innovative ideas for my games, Here is the list I came up with:

Idea List

Web Dev Simulator
Hole Skating

Web Dev Simulator

You are a website developer. You have a budget You need money OK. Enough small talk; GET WORKING!!!

I intend this to be sort of a joke, possibly a stab at the world of SEO and “Professional” website development. You might have to purchase SEO packs, and choose a web host from a randomly generated list of names. They might range from Free-and-Scammy to Expensive-and-Professional. Your host might have downtime, the search engines might lower your ranking, it’s all possible in this game. Also, the only reason I’m making a simulator is because of the simulator craze started by PewDiePie. Surgeon Simulator, I’m looking at you.

Hole Skating

You are lying in bed, and are suddenly woken up from your beauty-sleep by the sound on metal-on-ice. You rush out of bed and look through the window, and what do you see?

You see people who are skating on the ice. That’s irritating, you think, expecially as they have not taken your permission and it’s your backyard.

Wait a minute. How did your backyard freeze? THOSE IDIOTS MUST HAVE POURED WATER ON IT!!

You put on your sweater, your muffler and skates. You attach a salt shaker to the skates. Now, let’s teach them a lesson

Pardon me for making you read all that backstory, I got carried away :P

In this game, your objective is to create holes in the ice, and make other people fall through. Beware, the holes will eventually freeze over again, and the people will keep getting smarter. Don’t fall into the hole you thawed for others, though.

The End

What do you think about these ideas? Want to contribute resources for their development? Want to raze them to the ground? Let me know what you think

Benchmark public javascipt CDNs on a VPS

Sun, 20 Jul 2014 00:00:00 +0000

This is a legacy script from when I was 13 years old. Please don’t give me a hard time

#!/bin/bash
function speedTest {
	# ripped off from http://freevps.us/downloads/bench.sh 
	local the_speed=$( wget -O /dev/null "http://$1" 2>&1 | awk '/\/dev\/null/ {speed = $3 $4} END {gsub(/\(|\)/,"",speed); print speed}')
	echo $the_speed
}

echo "jsTest.sh by IshanDS www.ishands.cf"
echo ""
# CDNJs
cdnjs=$(speedTest "cdnjs.cloudflare.com/ajax/libs/jquery/2.1.1/jquery.min.js")
echo "Speed from CDNJs:             $cdnjs"

# Google CDN
google=$(speedTest "ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js")
echo "Speed from Google CDN:        $google"

# jsDelivr
jsdelivr=$(speedTest "cdn.jsdelivr.net/jquery/2.1.1/jquery.min.js")
echo "Speed from jsDelivr:          $jsdelivr"

# MaxCDN OSS
maxcdn=$(speedTest "oss.maxcdn.com/jquery/2.1.1/jquery.min.js")
echo "Speed from MaxCDN:            $maxcdn"

# Jquery CDN
jquery=$(speedTest "code.jquery.com/jquery-2.1.1.min.js")
echo "Speed from Jquery CDN:        $jquery"

# Microsoft CDN
microsoft=$(speedTest "ajax.aspnetcdn.com/ajax/jquery/jquery-2.1.1.min.js")
echo "Speed from Microsoft CDN:     $microsoft"

echo ""
echo "Thank you for using jsTest.sh"

Search

Mon, 01 Jan 0001 00:00:00 +0000

Welcome on Ishan Das Sharma

How to Generate Presigned URLs for R2 When Using Cloudflare Workers (And Make Sure They Actually Work)

Step 1: Use the Right Library

Step 2: Generate the Presigned URL

Step 3: The Content-Type Trap (This One Got Me)

Step 4: Configure CORS (The Most Important Part)

Step 5: Set Up Your Environment Variables

The Complete Flow

Testing Your Implementation

The Full Picture

Final Thoughts

What I Learned Using Neon Postgres for a Week of Live Traffic

What I Was Actually Running

The Good Parts

The Tradeoffs

Thoughts on Pricing

Serverless Databases Require Frontend Discipline

My Actual Usage

The Paid Tier Math

Would I Use It Again?

How to Handle "Bracket" Array Query Parameters (key[]=val) in FastAPI

The Code

Usage

How to Force Google Calendar Account Picker When Adding ICS Subscriptions

Use AccountChooser to Force the Picker

Why This Works

Make Your Own AccountChooser Link

When to Use It

Easy alias for dealing with Python Virtualenvs

I’m an IIT Madras Student. But to Some, I’m Diluting the Brand.

From “Cracking JEE” To Tarnishing “Made in India”

The Purpose Of A JEE Is Who It Selects

The Author Is A Platypus

Life As A “Brand Diluter”

The Psychology of Exclusivity

An IIT Is For Nation Building

A Message To The Gatekeepers

Easy Jigsaw Puzzle Embed for Your Website

What It Does

How to Add It

Detecting When the Puzzle is Solved

Try It Out

Preview:

Credits

Announcing a new chess bot - A0lite-js

What was a0lite?

Why TypeScript?

What is a0lite-js?

Current Limitations

Future Development

Getting Started

Contributing

Send Applescript Notifications from your CLI easily with this nifty alias

Usage

Why FileLinks Didn't Work in Google Colab

The Problem with FileLink in Google Colab

The Button-Based Solution

When Using `bc`, `scale` Doesn't Round!

The Initial Problem

What We Discovered

The Practical Solution

Key Takeaways

Read This Next: Using AI For Recommending Posts On My Blog

Why Move Beyond Traditional Recommendation Systems?

RAG, Text Embeddings and GPT-4

Summarizing Posts

Embedding and Storing

Finding Related Posts

Putting It All Together

Read Next

Calculating The Number Of Digits In A Power Of 2

Read Next

Blogging with Jupyter Notebooks in Hugo

This Is A Jupyter Notebook

Thinking about Options

Looking Inside Notebooks

But What About Images?

The ipynb Shortcode

Why a Shortcode?

Copy Me!

The Problem with `FileLink` in Google Colab

The `ipynb` Shortcode