Home Archives Search Feed

Consistent Hashing

In computer science, consistent hashing is a special kind of hashing technique such that when a hash table is resized, only n/m keys need to be remapped on average where n is the number of keys and m is the number of slots. In contrast, in most traditional hash tables, a change in the number of array slots causes nearly all keys to be remapped because the mapping between the keys and the slots is defined by a modular operation.

Basic technique

Consistent hashing was designed to avoid the problem of having to reassign every BLOB when a server is added or removed throughout the cluster. The central idea is, we use a hash function that randomly maps both the BLOB and servers to a unit circle, usually 2radians. For example, {= % 360} (where is hash of a BLOB or server’s identifier, like IP address or UUID). Each BLOB is then assigned to the next server that appears on the circle in clockwise order. Usually, binary search algorithm or linear search is used to find a “spot” or server to place that particular BLOB in O(N) or O(N) complexities respectively; and in every iteration, which happens in clockwise manner, an operation {  } (where is the value of the server within the cluster) is performed to find the server to place the BLOB. This provides an even distribution of BLOBs to servers. But, more importantly, if a server fails and is removed from the circle, only the BLOBs that were mapped to the failed server need to be reassigned to the next server in clockwise order. Likewise, if a new server is added, it is added to the unit circle, and only the BLOBs mapped to that server need to be reassigned.

uid: 202208200937 tags: #software-engineering #algorithms

February 22, 2023

Figure Out Which Crud Requests Need To Be Made Resilient That Currently Aren’T.

Map async

CRUD requests that use liveShardsHelper.mapAsync:

  1. Users get most recently viewed (?) — not application scoped
  2. Users add favorite (?) — not application scoped
  3. User enroll workspaces in block limits experiment — not application scoped
  4. Application read sync activity log for multiple views — ✅
  5. Workspace undestroy (?) — not application scoped
  6. Enterprise account get share infos for report — not application scoped
  7. Enterprise account get page bundles — not app scoped
  8. Enterprise account get applications — not app scoped

Also, we have getPublishedPageBundlesForDisplayAsync, which is a “crud live shards util”. This is used in:

  1. User list applications and page bundles for display — not app scoped

Another “crud live shards util” is getPageBundleSharingIntentsAsync. This is used in:

  1. User get page bundle ids shared via intent — not app scoped

Run on shard for application ID async

CRUD actions:

  1. Workspace move application — not app scoped
  2. Workspace add application as clone — not app scoped
  3. Crud user utils — _isWorkspaceOrApplicationOrPageBundleActiveAndCanShareWithUserAsync. This is used in several other methods in crud user utils, but ultimately circles up to user accept invite and user verify email. — not app scoped

runOnAllShardsInParallelAndExcludeShardTransferRemnantsAsync

  1. Block installation read helpers — https://github.com/Hyperbase/hyperbase/blob/33d91eabe2e2e2cefbecad956bb1d62b45b2e40e/worker_service/crud_managers/block_installation_read_helpers.tsx#L294
  • app scoped crud requests here:
    1. Application add table as clone ✅
    2. Block installation read ✅

runReadonlyActionOnShardForApplicationIdAllowingMultipleShardsPerDomainAsync

  1. Application undestroy block installation - https://github.com/Hyperbase/hyperbase/blob/33d91eabe2e2e2cefbecad956bb1d62b45b2e40e/worker_service/crud_managers/crud_application_action_handlers/private_api/application_undestroy_block_installation.tsx#L204 ✅
  2. Create app as clone from db, which is called in
    1. Workspace add application as clone — not app scoped
    2. Workspace add application from share — not app scoped
    3. Workspace add application as template clone — not app scoped

Created from: Journal entry: 08-17-22 202208170801

Scratch

  1. Application add collaborators — requires a write to main, canonical record for app collaborators is on the main shard

uid: 202208171607 tags: #airtable

February 22, 2023

Describe a Version of Yourself That Seems Impossible Right Now

Inspired by this article. If you haven’t read it already, go ahead and read it. It’ll be worth it, trust me.

1. It seems impossible that I’ll be someone who has an established online presence. Someone who writes regularly, has people following his work, and has found an online scenius of like-minded people.

Nothing else comes to mind as obviously impossible right now, which I think says a lot about the progress I’ve made since the last time I (informally) did this exercise a couple of years ago:

  1. At the time, being someone who worked out consistently (and could see the benefits of doing so) felt unattainable. While I haven’t worked out consistently over the past year, I have achieved the vital goal of making exercise an integrated part of my lifestyle.
    1. I’ve been walking roughly 8000 steps a day for the past 2+ years.
    2. I trained for and ran a 5k. I also trained for a 10k, although I got COVID and wasn’t able to run it.
    3. I’ve finally found an exercise routine that I’ve managed to stick to for a consistent period. I feel excellent after I work out, and I’ve learned about and invested my time/money/attention into crucial supplementary concepts like nutrition and sleeping well.
      1. I’ve seen minor qualitative improvements in my physical appearance and weight PRs. Two of my goals for this upcoming year are to 1) work out consistently enough so that these improvements are major instead of minor and 2) track data in this base so I can see those improvements qualitatively.
  2. I didn’t think I was someone that people truly cared about in social settings. I felt insecure about my life and didn’t think most of it was worth sharing with other people. This is no longer true — I now feel very comfortable about my social worth and don’t feel FOMO or jealousy when I see other people thriving in social situations. (There are many reasons for this change, but probably the most important one was that I lucked into finding a fantastic group of friends who appreciated me and with whom I could be completely myself.)
    1. I still don’t share much of myself online, but I plan to change that with this blog!

uid: 202207101318 tags: #blog-ideas #completed #living-well

February 22, 2023

# I should have loved biology

source: https://ift.tt/IxP9UAC tags: #favorites #literature uid: 202207101309 —

But biology, like computing, has a bottom, and the bottom is not abstract. It’s physical. It’s shapes bumping into each other. In fact the great revelation of twentieth-century molecular biology was the coupling of structure to function. An aperiodic crystal that forms paired helices is the natural store of heredity because of its ability to curl up and unwind and double itself with complements. Hemoglobin, the first protein studied in full crystallographic detail, was shown to be an efficient store of energy because of how oxygen atoms snap into its body like Legos, each snap widening the remaining slots, so that it loads itself up practically at a gulp. Most proteins are like this. The ones that drive locomotion twist like little motors; the ones that contract muscles climb and compress each other. Cells, too, are constantly in conversation, and the language they speak is shape. It’s keys entering locks: a protein might straddle the cell membrane, and when a cytokine (that’s a kind of signaling molecule) docks with it, it changes its shape, so that its grip loosens on some other molecule on the interior side of the membrane, as though fumbling a football—that football might be a signal itself, on its way to the nucleus.

.insights Beautiful paragraph. #insights

For a computer scientist, a biologist’s methods can seem insane; the trouble comes from the fact that cells are too small, too numerous, too complex to analyze the way a programmer would, say in a step-by-step debugger. What biologists mostly do is stuff like:

Spin things to 15,000 Gs in centrifuges to separate pieces having different densities.
Separate things of different sizes using gels and magnets. (“Gel electrophoresis.”)
Take one of those gels and blot it with special paper to splay the parts out. Then wash the paper with an antibody that binds to a specific protein. Finally, wash the paper with another antibody that binds to the first one, and fluoresces when it does so. See where the meta-antibody lights up—that’s the protein you were looking for. (I think I’m describing a “Western blot.”)
Use the fluorescent antibody trick to tag cells expressing one or more proteins of interest. Then squeeze the cells through a tube so small that only one fits at a time. As each cell passes by, shine a laser through it to read its fluorescent tags, and use an electric charge to redirect it to a particular bin. Now you can sort and count cells that match your criteria. (“Flow cytometry.”)
Genetically alter microorganisms to make molecular machines to spec; systematically turn off one gene at a time in a cell line and see what changes; edit the genome of a whole animal, and observe its life.

February 22, 2023

Stripe Application

Referencing Nate’s comments here: 202009140035

If there’s one thing I’ve learned from my internships, it’s this—I’m most productive and engaged with my work when my own morals and values are completely in sync with the company’s culture. Everything else follows from those first few well-defined principles.

I feel that way about Stripe’s values—as someone whose two current biggest goals in life are to improve every day and be a good person, I see how I would be totally at home with “Trust and amplify” and “Global optimization” as guiding directives at my workplace. I’m especially enamored by “Move with Urgency and Focus” as a core value—in my prior internships, I’ve sometimes failed to seek feedback early, being hesitant to push out unfinished work, and only realized later how much smoother things would’ve been if I had iterated quicker. I’m really impressed that Stripe has identified the importance of that and made it a company-wide cultural mandate, so no one feels afraid or awkward to ask for feedback when their work isn’t quite finished. To me, that’s the sign of a company that tries as hard as it can to create a productive and fulfilling workplace for its employees.

Another big reason I want to join Stripe is because of Stripe’s unique position as a startup yet industry leader — I think this opens up a huge number of very interesting, very specialized problems (that haven’t been solved yet!) that I would be able to try tackling as a new grad. I think the range of these problems wouldn’t be too far out of my wheelhouse (roughly), but still new enough to me that I’d be growing leaps and bounds as a collaborator and engineer.

I’m also a big fan of the attention to detail (I love the font and color scheme on the website!) and the unapologetically developer-focused experience that Stripe delivers.

uid: 202010061005 tags: #recruiting #applications #archived

February 22, 2023

Comparing Theorem Vs Amazon

With Theorem, I dived right in - I just got my hands dirty. I got the chance to work on immediate, pressing problems that were critical to both the current success and future growth of the company. I was collaborating directly with full-time engineers. I was developing in-house tools and infrastructure that directly addressed the specific needs the company had at that time.

With Amazon, I got the chance to experience working with technologies at scale, and making both high-level design decisions and low level implementation choices that I definitely wouldn’t have had the opportunity to make at a smaller company. Or arguably, any other company, given Amazon’s unique combination of scale, frugality, and customer centricity.

At Amazon, one challenging aspect was that there was an absolute overload of information and resources, and sometimes it was difficult to parse out the information that was relevant to the task at hand. Like in the internal wiki for each common tool, there would usually be about 4-5 separate articles, written by different people, each detailing their own perspective on theirs and cons of the cool. Similarly, in the internal stack overflow, there would be several answers for any given question that you would search, and even worse sometimes there would be 3-4 different variations of the same question, which would make you question whether you actually understood your specific problem or not. So that was definitely a huge learning opportunity for me, and by the end of the internship, I felt much more confident about parsing through complex wikis and documentation and being able to apply that information to my specific task at hand.

uid: 202007222351 tags: #theorem #amazon #2020recruiting

February 22, 2023