CAPULET: Cache Pooling Metadata Caches in Secure Disaggregated Memory Systems

Or, more generally, secure memory through CXL

Compute Execute Link (CXL) is a protocol that allows CPUs to access memory through I/O ports. Traditionally, CXL has been used to connect CPUs to external DRAM. A CXL system’s memory capacity is not limited by the number of memory ports, as the connecting switches can be attached to more switches. Therefore CXL is extremely scaleable and well-addresses the RAM shortage caused by the AI boom. Furthermore, the only additional latency introduced by CXL is the physical link itself. Our research proposes that CXL also has the potential to connect CPUs to external SRAM. CAPULET (Cache Pooling Metadata Caches in Secure Disaggregated Memory Systems) is a protocol in which multiple CPUs can share their cache resources through CXL while also maintaining data integrity with a Binary Merkle Tree (BMT).

The primary goal of CAPULET is to reduce the miss rate of BMT lookups (i.e., to reduce the likelihood that the data we are looking for is no longer in cache) by letting multiple hosts pool their cache resources. CAPULET keeps track of the least-recently used block, which is where evicted data will go. Theoretically, this will mean that the cache will be filled with the most-likely-to-be-reused data. Some additional advantages of CAPULET are:

1. Cryptographical security through BMT hashing A Binary Merkle Tree (BMT) is a hash tree in which each leaf node contains the hash of a block of data (e.g., hash-based message authentication code (HMAC)), and every parent node holds the hash of the concatenation of all its children’s hashes. This means that the root node is a hash that encodes every child hash. And, in the case that the root node can be trusted, then we can use it to verify the integrity of all other nodes in the tree. (To ensure that the root node can be trusted, we store it on-chip.) This means that, if we keep record of which parent nodes we trust, we can significantly reduce the number of hash lookups necessary to verify any given block.
2. The inherent security of SRAM over DRAM Unlike DRAM, SRAM is not vulnerable to rowhammer attacks. This is because SRAM, unlike DRAM, does not leak charge and therefore does not require periodic refreshing. This is very important as the increasing demand for memory capacity means that the typical cell density of DRAM devices is also increasing, and cell density is directly related to how vulnerable a device is to rowhammer attacks.
3. Faster Fetch Times As CAPULET is a cache-centric protocol, naturally we will take advantage of the fact we do not need to fetch from DRAM as often. This means that the overall fetch times, and therefore the overall runtimes, will be faster.

The CAPULET emulator

For our research we used a python emulator and packet traces from running SPEC CPU 2017 benchmark workloads on gem5. There are three main classes in the emulator: Host (representing a single host CPU), Cache (which manages the local caches), MetadataCache (which manages the BMT system), and CAPULET (which is a wrapping class that orchestrates all of the other classes).

The emulator starts by reading the packet trace files in a specified directory (which we filled with traces of SPEC CPU 2017 benchmarks) and then feeding them into the CAPULET class, which then creates the specified number of hosts, each of which is given its own disjoint address range. Each host is then given a MetadataCache. The CAPULET class then progresses the traces/hosts based on their timestamps. The host with the smallest timestamp goes next. Once a host reaches the end of its current trace, CAPULET assigns it another trace. The emulator ends as soon as every host has completed at least one trace. (This is because trace lengths vary significantly, and the largest trace is dozens of times longer than the smallest trace)

The heart of CAPULET, the eviction-broadcast system, is in Cache.fill(). At every eviction, there is a p probability that the evicting host will broadcast a message to the other hosts. To find the ideal balance between messages sent vs. hit rate, we varied p throughout our simulations.

Results and limitations

Currently we are in the stage of running simulations with the SPEC CPU 2017 benchmark suite. The simulations vary in the number of hosts and eviction-message broadcast rates. Preemptively, we are getting overall hit rates of 97% to 99% for most hosts, and metadata hit rates generally increase as the rate of eviction-messages increase.

A potential limitation to note is that, in our simulations, all CPUs are running simultaneously, which may not necessarily be reflective of a real-world scenario. Furthermore, the timestamp density of the packet traces varied significantly, which may further exacerbate the prior point.

Next steps

I would be very interested in modifying the eviction-broadcast messages so that they are more intelligent than our current p-system. For example, I am wondering if restricting broadcasts to a select group of “sister hosts” would reduce excessive messages, or if I could make p dynamically adapt to each host’s current demand. The final data of our p-experiments will be a very helpful guide for future development.

Closing notes

This project continues what my advisor, Prof. Sam Thomas, started during his grad school program. I was given a mostly-complete emulator and packet traces to start working with. In the first weeks, I spent time debugging the emulator and making sure it performed as expected. I found and fixed issues with the disjoint addressing (how the CAPULET emulator decides which virtual address ranges represent which host) and scheduling (how the CAPULET emulator decides which trace/host progresses next). A major challenge was figuring out how to normalize the “current timestamp” of each host as the starting timestamps and timestamp densities both varied significantly. I also adapted the emulator so that each host could run a unique workload. The original emulator had it so that each host would run the same trace file.

On a more personal note, I am very grateful that I got to experience this new side of computer science. It was amazing to see just how well-connected the computer architecture community is when I attended ISCA this summer. I would also like to thank my labmates Arsene Laurent and Vadym Musiienko, and my now-graduated labmate Stevie Kim, who was a role model not only in academics but also in Oldenborg’s Japanese hall. And of course, special thanks to Prof. Thomas for this incredible opportunity and support he provided, both academic and moral.

Mirakuru: Software-Hardware Co-Design for Highly Concurrent Heterogeneous CXL Lock-Free Data Structures

The Problem

DRAM is one of the largest and fastest-growing costs in the datacenter. A DIMM’s useful lifetime exceeds a typical server refresh cycle, yet it is decommissioned with the host. And DIMM slots are designed for the most current DDR generation, stranding functional older modules. The result is stranded capacity and avoidable embodied carbon. A state-of-the-art solution to this financial and environmental cost is Compute Express Link (CXL), which enables flexible memory expansion and the reuse of decommissioned DIMMs. A standard setup exposes a remote pool of heterogeneous memory to the operating system as a CPU-less (zNUMA) node, shown in the figure below.

A zNUMA node is a NUMA node that carries memory but no CPU of its own, so the host reaches across CXL purely to load and store data in that pool.

Why Lock-Free Structures Struggle on CXL

However, CXL memory is slower than local DRAM, so concurrent data structures placed on it, such as hash tables and skip lists, can degrade an application’s quality of service (QoS). These structures rely on compare-and-swap (CAS), which updates a shared location only if it still holds the value the thread last read.

Here is what that means in practice. A thread reads a shared location, decides what it wants to write, and then asks the hardware to write the new value only if the location still holds the value it read a moment ago. If another thread got there first and changed it, the CAS fails and the losing thread has to start its operation over.

Figure below illustrates how such an update can fail, forcing thread 1 into its fallback path, where it must redo its find(B) call to recover the true prev and curr. Picture two threads working on a linked list. Thread 1 walks the list to find where its new node belongs, landing on a prev and a curr. Before it can splice its node in, Thread 2 deletes curr, so prev no longer holds what Thread 1 expects, and its CAS fails.

CXL’s higher latency extends the contention window between a thread’s read and its update, letting more threads collide on the same data. The two heatmaps below (local memory and conventional CXL) show that CAS success on the slower, remote CXL memory falls below local by up to 8.4% in the most contended configurations, so non-blocking applications hit their fallback paths more often. Every one of those failures is a thread throwing away work and redoing it.

The Idea: Expose the Heterogeneity

How often an object’s CAS succeeds depends on which device holds it, because the pool mixes devices of different latencies. Yet the CXL interface as constructed today is needlessly restrictive. By presenting the pool as uniform memory, it prevents the programmer from placing the most contended data on the fastest device. Our proposed co-design exposes the heterogeneity the interface hides.

The rest of the work is making that heterogeneity visible in a way software can act on, and it starts down in the memory controller.

Building the Controller in gem5

At boot, our CXL controller initializes the connection to each attached device; the handshake reports that device’s timing characteristics, from which the controller derives a priority ordering over the pool, fastest first. It then maps physical addresses to devices on demand: when a request arrives for an address that has not been mapped yet, the controller binds that address to the fastest device with capacity remaining.

A few details are worth spelling out. Because the ordering comes straight from the boot handshake, the controller already knows that DDR5 beats DDR4 beats DDR3 before a single memory request arrives. The mapping is sticky: once an address lands on a device it stays there. Addresses are bound in the order they are first touched, not by their numeric value, so the earliest-touched addresses get the fastest memory and later ones fall through the tiers in speed order as each device fills. That is what later lets software reason about placement: an allocation that reserves the pool and walks it in order sees the devices laid out fastest-first, so a device’s speed rank corresponds to a known slice of the reservation.

The controller ships with three placement strategies, selected by a parameter. direct passes physical addresses through to the pool untouched, which is what a conventional CXL controller does. random assigns each newly touched cache line to a random device, our baseline. speed is Mirakuru’s fastest-first mapping described above.

The Controller as a Simulated Device

The controller lives in three files. src/mem/CXLcontroller.py declares its parameters to the configuration layer, src/mem/cxl_controller.cc holds the C++ that does the work, and src/python/gem5/components/memory/cxl_memory.py wraps it in a memory component that any board can accept in place of ordinary DRAM. The board attaches to the controller and sees one flat range of memory. Behind it sit the individual devices, each with its own DRAM timing model, and the board never learns they exist.

gem5 does not have one way of accessing memory. It has three, and a full-system run uses all of them, so the controller has to implement all three.

• Functional accesses are used to set memory up: loading the kernel and disk image, and debugger reads. They are not supposed to take any simulated time.
• Atomic accesses serve the fast-forward cores that boot Linux. There is no scheduling here, so the call has to return a latency on the spot.
• Timing accesses are the real ones, used once the region of interest switches to cycle-accurate cores. A request is sent now and the response comes back later.

All three share the same address translation. What differs is how packets are handled, and that difference is a trade-off. Functional and atomic accesses are synchronous, so the controller rewrites the address on the caller’s own packet and passes the same object along, which costs nothing. Timing requests outlive the call, so the controller has to allocate a copy carrying the device address, keep a record tying it back to the original, and reunite them when the response arrives. Copying every packet is real overhead, but only the timing path pays it, and the timing path is the only one whose numbers I report.

Controller Latency

CXL memory is slower than local DRAM, and the controller has to add that delay itself. Its latency is a parameter, exposed up to a command-line flag on the config script, so sweeping it needs no recompile. It is specified as a round trip and applied half on the way to memory and half on the way back, so the number in the config is the number you would measure on a real link.

In timing mode there is nothing to return a latency to, so the controller schedules. It holds requests and responses in queues stamped with the tick at which each is allowed to leave, and wakes itself at that tick to pass the packet on. If the component downstream is busy and refuses a packet, the controller waits to be told to retry rather than spinning, and once unblocked it releases anything whose deadline has already passed.

The queues are unbounded, so the controller never pushes back on the CPU no matter how much traffic arrives. That is a deliberate simplification: it models the link’s delay and not its bandwidth. This study is about how latency stretches the contention window between a thread’s read and its CAS, so leaving bandwidth out keeps one variable in the experiment instead of two.

Fragmentation

The direct strategy models a conventional CXL controller, and on a freshly simulated pool it flatters one. Pass physical addresses through untouched and a contiguous allocation lands contiguously, so the benchmark’s array sits on whichever device happens to back that stretch of the address space, and often on that one device alone. It never spans the pool, so it never sees the heterogeneity the study is about. Real pools do not look like that. Memory is allocated and freed over a machine’s life, and a large allocation ends up spread across whatever holes are left, spanning devices. Without fragmentation the baseline is not a baseline.

My first version produced it honestly, from inside the benchmark itself. Before touching the CAS array, the program allocated the entire pool in small blocks, wrote to each one to force the controller to map it, then freed a configurable percentage at random. Whatever the benchmark allocated afterwards fell into those holes, the same way a real allocator would be forced to.

It worked, and it was far too slow. Every one of those writes is a simulated store. Touching the whole pool a small block at a time means the benchmark drives the entire memory through the controller, one cache line at a time and all of it through the full timing model, before it performs a single real operation. Runs that should have taken minutes did not finish.

The fix was to notice what I actually needed. The guest never had to produce a fragmented layout. The controller only had to start from one, and it can build that in host code at construction, before any simulated time passes.

So I deleted the benchmark-side version and made fragmentation a property of the controller. It divides the pool into fixed-size granules, picks a random subset of them, and permutes their addresses among themselves. The percentage of granules disturbed is a parameter: at 100 the entire pool is shuffled, at 40 a randomly chosen 40% trade places while the rest stay put, and at 0 the controller skips the work entirely and passes every request straight through untouched. A seed parameter fixes the permutation so a sweep over thread counts and array sizes compares like with like. On the request path the whole mechanism reduces to one array lookup.

The trade-off is that this models fragmentation rather than reproducing it. A real system fragments at page granularity, through the interaction of a kernel allocator, a user-space allocator, and a workload’s allocation history. Mine imposes the scatter from below, at a granularity I choose. It gives the experiment the property it depends on, which is that a contiguous run of physical addresses lands on devices of different speeds in proportions I control. It does not claim to reproduce any particular allocator’s behavior.

A fragmented run now costs the same wall-clock time as an unfragmented one, because the controller does the same amount of simulated work in both.

The Address Map

The random and speed strategies pick a device the first time an address is touched and have to remember that choice for the rest of the run. The obvious structure is a hash map: constant lookup, constant insert, and memory proportional to what the workload actually touches rather than to the size of the pool. That is what I wrote first.

It turned out to be the wrong choice. The map is consulted on every memory access in the simulation, and it grows to millions of entries as the guest’s footprint expands. Constant time here hides a hash, a modulo by a load factor, a bucket lookup, and a pointer chase through a structure far too large to sit in any host cache, plus a rehash of everything each time the table outgrows itself. So the simulation got slower as the pool filled, and the slowdown was worst in the large-pool configurations I most wanted to measure.

I replaced it with a flat array. The pool’s size is known before the simulation starts and every request is cache-line aligned, so the map can be one array slot per cache line, allocated once and pre-filled with a sentinel meaning “not yet assigned.” An index computation replaces the hash, and a lookup becomes one memory read with no indirection.

The trade-off is memory for speed, paid up front. The array is sized for the entire pool whether the workload touches all of it or none of it, which costs about 12% of the pool’s size in host RAM. A hash map would have used a fraction of that for a sparsely touched pool. I took the memory hit because simulation wall-clock time was the binding constraint on how many configurations I could sweep, and host RAM was not.

One more detail keeps the array small: it stores one entry per cache line rather than one per address. The low bits of an incoming address are split off before the lookup and added back to the device address afterwards, so a narrow request still lands on the correct byte without the map ever holding an entry for it.

Fixing the x86 Board

Both maps index by address, and on x86 that turns out not to be a matter of dividing by the block size, because the guest’s physical address space has a hole in it. The region between 3 GiB and 4 GiB is reserved for memory-mapped I/O, so on a machine with more than 3 GiB of RAM the memory sits on both sides of a gap that nothing backs. Indexing naively would have sized my array for the hole as well and wasted a slot for every address in it. Instead the controller walks its device ranges and collapses them into a dense index space, so the map has no entries for addresses no device serves. The same routine runs in reverse to turn an index back into a real device address, which is what the fragmentation shuffle uses, and both maps share it by passing in the granularity they care about.

Getting that far meant fixing gem5 itself. X86Board, the board model for full-system x86, refuses to be configured with more than 3 GiB of memory. It throws an exception and names the I/O hole as the reason, which is a hard ceiling for a project about large memory pools.

The board was closer to working than the exception suggested. I removed the ceiling in src/python/gem5/components/boards/x86_board.py and taught the board to split memory around the hole, putting everything up to 3 GiB below it and the remainder above 4 GiB. Two further things were broken underneath, both of which had to be fixed before a larger machine would boot and see all of its memory.

None of this is CXL work, but the fix is general: any gem5 user who wants a full-system x86 machine with more than 3 GiB of RAM needs it.

Testing It: The Benchmark

We evaluate the co-design on a highly concurrent array of atomic elements on which we perform CAS operations.

We implement the controller in gem5 and run the benchmark in full-system mode on a simulated x86 Linux host with 64 cores. Its cores are cacheless and it uses no local memory, so every access reaches the pool, isolating the controller’s impact. The pool comprises three devices, fastest to slowest: DDR5, DDR4, and DDR3.

The cacheless, no-local-memory setup is deliberate: with no cache to hit and no fast local read to fall back on, every mapping decision the controller makes lands in the numbers. We sweep two knobs, the thread count and the array size, because their ratio sets the level of contention. We also draw accesses from a Zipfian distribution, so a small set of hot elements absorbs most of the traffic, the way real keys do.

As an upper bound, we run the same host with a single local DDR5 device. Our baseline is the same pool under random placement, assigning each cache line to a random device.

Results

Figure below reports the CAS success rate under our fast-priority CXL controller. We define the high-contention cases to be those in which the number of threads exceeds the size of the array, where placement matters most. Across them, the controller improves the success rate by as much as 7.1 and by 4.2 percentage points on average over the baseline, a relative gain of up to 25.5%. From this we conclude that CXL-awareness can improve application performance.

Read the three heatmaps together: local DDR5 sets the ceiling, conventional CXL sits well under it in the high-contention corner, and Mirakuru pulls the pool back toward local without any help from the application yet.

What’s Next

As future work, we propose a topology-aware allocator built on this controller. It reserves the entire pool in one contiguous allocation, and because the controller maps fastest-first, each device occupies a known, speed-ordered range within it. Our malloc then takes a size and a target device by speed rank, serving the block from that device and falling back to the next fastest when it is full. This gives the programmer the device-level placement the standard CXL interface hides, keeping hot data on fast memory and cold data on slow.

We also plan to deploy Mirakuru in the state-of-the-art non-blocking concurrent indices of the Synchrobench suite. Presampling the Zipfian key distribution, we place the most popular keys on the fastest device, keeping the hot set on fast memory and the cold tail on slower devices. The work in this study provides strong motivation that this approach will improve their performance as well, especially under contention-aware placement.

Project Title: Scr Mmy: A Secure Compressed Co-Designed Memory

As the world increasingly uses more and more RAM in daily life, it is pivotal that the information remains secure. However, the methods that are currently available to ensure this come at a cost of time. Secure Memory, a method that uses merkle trees to verify the data is secure in RAM, while proven to accomplish the security, has also been proven to take too long to be able to be used in production.

To solve this problem, our focus moved away from creating a faster method, as security is inherently slow, and instead focused on adding other features into the process that are able to make the slower speed a viable option. The first approach that we began looking into this summer was compression. We aimed to create an architecture that takes a secure memory architecture and co designs it with a memory compression architecture.

The memory compression that we are using as the basis for our compression architecture is DyLeCT compression. We chose this architecture as it uses similar structures that the secure memory uses. Because of this, we can continue to use the memory controller as the main component, and build it in a way that allows for both compression and security.

The work that we did this past summer was the foundation for the direction of the research. I worked to understand the needs of the secure system as well as the different compression systems that have been published. Designing the theoretical model, I began working to recreate the architecture from DyLeCT, using gem5, in a way that would integrate with the Secure Memory. Moving forward in the fall, I will continue to implement the two architectures together to be able to run analysis on the theoretical model.

gem5 Parallelization

Overview

This project explores how to parallelize gem5 through microservicing. The original gem5 implementation executes events sequentially through the serviceOne() function, which both removes the next event from the event queue and then executes it. As a result, each event is serviced one at a time. The goal of this project is to separate the process of popping events from the queue and executing the popped events. Ideally, this split will allow multiple worker threads to pull events from the queue and process them in parallel.

Approach

Our approach to parallelize gem5 is centered around multiple worker threads executing events at the same time as long as the events do not interact with the same simulated hardware component. To do this, the event queue logic is split into two steps, where one step pops the next event from queue and the second step executes the event. Since multiple threads could run at once, the implementation needs to prevent two threads from modifying the same hardware state at the same time. To handle this, we create a lock for each event->name() because in gem5 the event name corresponds to the hardware component associated with the event. The worker threads share the same event queue, with a global queue lock to protect the process of checking the queue. The lock per event protects the actual execution of each event. In theory, this means that multiple events can run in parallel as long as they are associated with different hardware components.

Progress

The current implementation does not work. It fails the assert (when >= tick) in schedule and reschedule. When commenting these asserts out, the functions do not run properly.

To get gem5 to run properly, edit simulationWorker in simulate.cc.

The working code that calls serviceOne is commented out, so uncomment this and comment out the calls to popNextEvent and executePoppedEvent.

The functional lines have a comment before them that says:

// this is WORKING simulate.cc line

The test lines say:

// this is for split serviceOne events (DOESNT WORK)

src/sim/eventq.hh

Lines 863 and 865 — function declarations:

// remove head event
Event *popNextEvent();

// does everything that serviceOne would do
Event *executePoppedEvent(Event *event);

src/sim/eventq.cc

Lines 55–56 — create a dictionary for a lock per event and a lock for the table as a whole. This is meant to prevent multiple events from trying to create locks at the same time.

std::mutex mutexTableLock;
std::unordered_map<std::string, std::mutex> eventLocksTable;

Lines 58–66 — retrieve or create a mutex for a given event:

std::mutex &getEventLock(const std::string &name)
{
    // unlock guard when done adding event to map
    std::lock_guard<std::mutex> guard(mutexTableLock);
    auto result = eventLocksTable.try_emplace(name);
    return result.first->second;
}

Lines 293–294 — acquire the event’s mutex and make it unlock once the event is done running:

std::mutex &eventLock = getEventLock(event->name());
std::lock_guard<std::mutex> eventGuard(eventLock);

Lines 257–279 — first half of serviceOne, the part that manages the queue:

Event *
EventQueue::popNextEvent()
{
    if (head == NULL)
        return nullptr;

    --queuedEvents;

    Event *event = head;
    Event *next = head->nextInBin;

    event->flags.clear(Event::Scheduled);

    if (next) {
        next->nextBin = head->nextBin;
        head = next;
    } else {
        head = head->nextBin;
    }

    setCurTick(event->when());

    return event;
}

Lines 281–309 — second half of serviceOne, the part that executes the event’s process:

Event *
EventQueue::executePoppedEvent(Event *event)
{
    if (!event) {
        return NULL;
    }

    if (!event->squashed()) {
        // setCurTick(event->when());

        if (debug::Event)
            event->trace("executed");

        currEvent = event;

        std::mutex &eventLock = getEventLock(event->name());
        std::lock_guard<std::mutex> eventGuard(eventLock);

        event->process();

        currEvent = nullptr;

        if (event->isExitEvent()) {
            assert(!event->flags.isSet(Event::Managed) ||
                   !event->flags.isSet(Event::IsMainQueue)); // would be silly

            return event;
        }
    } else {
        event->flags.clear(Event::Squashed);
    }

    event->release();

    return NULL;
}

src/sim/simulate.cc

Lines 174–178 — struct that holds the event queue, mutex to protect access, and atomic flag to tell worker threads when to exit:

struct multiThreads {
    EventQueue *eventq;
    std::mutex lock;
    std::atomic<bool> stop{false};
};

Lines 180–234 — worker thread main loop.

This attaches the thread to the event queue and keeps pulling events and running them. It locks the queue while checking if it is empty, then unlocks before executing. It stops once it hits an exit event.

static void
simulationWorker(multiThreads *shared, int worker_id)
{
    std::cout << "Worker thread " << worker_id << " running..." << std::endl;

    // Found in eventq.hh; sets the thread-local pointer to the current event queue.
    // This is for single thread, when eventq is a parameter to the function.
    // curEventQueue(eventq);

    curEventQueue(shared->eventq);

    // memory order relaxed is
    while (!shared->stop.load()) {
        Event *event = nullptr;

        {
            // take event queue lock
            std::lock_guard<std::mutex> guard(shared->lock);

            // std::cout << "Work being done by" << worker_id << std::endl;

            if (shared->eventq->empty()) {
                // std::cout << "Worker thread " << worker_id
                //           << " sees an empty event queue." << std::endl;
                break;
            }

            // Optional full queue dump while debugging.
            // shared->eventq->dump();

            // this is WORKING simulate.cc line
            // event = shared->eventq->serviceOne();

            // this is for split serviceOne events (DOESNT WORK)
            event = shared->eventq->popNextEvent();
        } // queue lock is released

        // testing with split serviceOne events (DOESNT WORK)
        Event *exit_event = shared->eventq->executePoppedEvent(event);

        if (exit_event != nullptr) {
            shared->stop.store(true);
            break;
        }

        // this is WORKING simulate.cc line
        // if (event != nullptr) {
        //     shared->stop.store(true);
        //     break;
        // }

        // this is for single thread when eventq is a parameter
        // while (!eventq->empty()) {
        //     Event *event = eventq->serviceOne();
        //     if (event != NULL) {
        //         break;
        //     }
        // }

        std::cout << "Worker thread " << worker_id << " done." << std::endl;
    }
}

Lines 238–241 — starts thread using a helper function:

static std::thread
startSimulationWorker(multiThreads *shared, int worker_id)
{
    return std::thread(simulationWorker, shared, worker_id);
}

Lines 320–339 — sets up worker threads, starts the workers, and joins them once they finish:

const int num_workers = 1;

multiThreads shared{getEventQueue(0), {}, false};

// Dynamic array of thread objects; eventually pushes each worker to this.
std::vector<std::thread> worker_threads;

// Capacity for the number of workers.
worker_threads.reserve(num_workers);

for (int i = 0; i < num_workers; i++) {
    int worker_id = next_worker_id.fetch_add(1);

    // New element to end of vector, stores each thread.
    worker_threads.emplace_back(startSimulationWorker(&shared, worker_id));
}

// For each element in worker_threads, call each thread t.
for (std::thread &curr_thread : worker_threads) {
    if (curr_thread.joinable()) {
        curr_thread.join();
    }
}

Project Title: CXip List: A CXL-Aware Skip List

Description

Modern workloads require systems that have access to both large and fast memory. However, developments in memory systems lag behind developments in compute, leading memory to be the primary bottleneck of such systems. One of the state-of-the-art solutions that can provide both large and fast memory is Compute Express Link (CXL), an open interconnect standard that allows for the attachment of remote memory modules to a system via IO, offering high bandwidth and low latency. Each memory module has their own latency, and thus systems that incorporate CXL have a memory topology where certain remote memory modules have lower latencies than other. Thus, to design a performant system for CXL systems, applications should use this topology for optimizations.

This leads us to our main problem: current interfaces cannot distinguish between individual remote memory modules. Instead, they view all remote memory modules as one large node, as shown by the image below.

Since we cannot distinguish between individual remote memory modules, we cannot design systems around the unique memory topology that CXL provides. Thus, applications running on CXL systems will have inefficiencies, as it is highly likely that memory accesses occur across individual remote nodes, which is proven to be unoptimal by prior literature.

Thus, this project has two research goals: (1) to create a new interface in the gem5 simulator for CXL systems that distinguishes between individual memory modules, and (2) to create CXip List, a skip list that is optimized for CXL systems. We hope that this work will provide the necessary tools and guidelines that can enable others to develop more CXL-aware applications.

Current Progress: Stevie worked on this project throughout the Spring 2026 semester. He worked on implementing skip lists in C++, starting from a simple sequential implementation and moving to a concurrent implementation using locks. He also gave a 15-minute talk about this project at the SoCal Programming Languages and Systems Workshop in February.