LINQ Execution, Projection, Grouping, and Performance

Learning outcomes

Translate a reporting requirement into a readable pipeline, distinguish deferred execution from streaming, choose a materialization boundary, and replace repeated membership scans without changing equality, duplicates, or order. All examples use finite synthetic in-memory data and independent .NET 10 console programs.

The runnable examples were verified in Release mode with .NET SDK 10.0.401 and runtime 10.0.12 on Windows x64.

Read the pipeline as a sequence of decisions

• Where decides which elements remain.
• Select creates the output shape for each input element.
• GroupBy partitions elements by a key.
• OrderByDescending establishes the main ranking; ThenBy resolves ties.
• Take limits results; ToArray executes the pipeline and stores its result.

These are Enumerable operations over in-memory objects. The same-looking expression over a database provider can have different translation, execution, and cost rules.

Worked example: top paying customers

Requirement: include only Paid orders, calculate revenue and paid-order count per customer, rank revenue descending, break ties by customer ID ascending, and return at most ten rows. The fixture deliberately contains cancelled and pending orders and a three-way revenue tie. Each Order in this exercise is non-null.

Paste each full listing into a separate .NET 10 console project's Program.cs. Do not combine the listings: each has its own entry point and supporting types.

using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;

public static class Program
{
    public static void Main()
    {
        Order[] orders =
        {
            new(1, 2, OrderStatus.Paid, 30m),
            new(2, 1, OrderStatus.Paid, 20m),
            new(3, 2, OrderStatus.Paid, 10m),
            new(4, 1, OrderStatus.Cancelled, 100m),
            new(5, 3, OrderStatus.Paid, 40m),
            new(6, 1, OrderStatus.Paid, 20m),
            new(7, 4, OrderStatus.Pending, 500m)
        };
        foreach (var row in TopCustomers(orders))
        {
            string revenue = row.Revenue.ToString("0.00", CultureInfo.InvariantCulture);
            Console.WriteLine($"Customer {row.CustomerId}: {revenue}, orders {row.OrderCount}");
        }
    }

    public static Summary[] TopCustomers(IEnumerable<Order> orders)
    {
        ArgumentNullException.ThrowIfNull(orders);
        return orders
            .Where(o => o.Status == OrderStatus.Paid)
            .GroupBy(o => o.CustomerId)
            .Select(g => new Summary(g.Key, g.Sum(o => o.Total), g.Count()))
            .OrderByDescending(x => x.Revenue)
            .ThenBy(x => x.CustomerId)
            .Take(10)
            .ToArray();
    }
}

public enum OrderStatus { Pending, Paid, Cancelled }
public sealed record Order(int Id, int CustomerId, OrderStatus Status, decimal Total);
public sealed record Summary(int CustomerId, decimal Revenue, int OrderCount);

Verified output

Customer 1: 40.00, orders 2
Customer 2: 40.00, orders 2
Customer 3: 40.00, orders 1

Trace the transformation

1. The cancelled order contributes nothing, even though it has a large total. Customer 4 has no Paid order, so produces no group.
2. Customer 1 has 20 + 20 = 40 across two orders; customer 2 has 30 + 10 = 40 across two; customer 3 has 40 across one.
3. Select projects each group into a Summary. The tied revenue makes ThenBy essential: the displayed customer order is 1, 2, 3.
4. ToArray is inside TopCustomers, so this method returns fully computed rows. It does not return a deferred query.

GroupBy is deferred but non-streaming: when enumerated, it buffers its source before yielding groups. Ordering must inspect its input before yielding the ranked result. Take(10) at the end therefore does not mean that only ten original orders are read. For this pipeline, n input orders and c paid-customer groups give an ordinary expected grouping-and-ranking cost on the order of O(n + c log c), with O(n + c) working storage; exact sorting optimizations can vary. The final array holds at most ten summaries.

References: Where, Select, GroupBy, ordering, and LINQ-to-Objects execution classification.

Deferred does not mean cached, streaming, or exception-free

A deferred query describes work that runs on enumeration. Where and Select can process elements as they arrive; GroupBy and ordering buffer data. Re-enumerating a query can redo work and observe changed source data. Some sequences are expensive or one-use, so do not assume every IEnumerable can be traversed repeatedly at negligible cost.

Argument validation may happen when an operator is called. In contrast, failures inside a deferred predicate or selector occur when that callback runs. The following program shows both without relying on a real database or network call.

using System;
using System.Collections.Generic;
using System.Linq;

public static class Program
{
    public static void Main()
    {
        var source = new List<Box> { new(1), new(2), new(3) };
        int calls = 0;
        var even = source.Where(x => { calls++; return x.Value % 2 == 0; });
        Console.WriteLine($"After declaration: {calls}");
        source.Add(new Box(4));
        Console.WriteLine($"First: {string.Join(", ", even.Select(x => x.Value))}");
        Console.WriteLine($"Calls: {calls}");
        source.Add(new Box(6));
        Console.WriteLine($"Second: {string.Join(", ", even.Select(x => x.Value))}");
        Console.WriteLine($"Calls: {calls}");

        Box[] snapshot = source.ToArray();
        source.Add(new Box(8));
        source[1].Value = 20;
        Console.WriteLine($"Counts: snapshot {snapshot.Length}, source {source.Count}");
        Console.WriteLine($"Shared element: {snapshot[1].Value}");

        try { _ = Enumerable.Where<int>(null!, x => x > 0); }
        catch (ArgumentNullException) { Console.WriteLine("Null source: immediate"); }
        var risky = new[] { 1, 0 }.Select(x => 10 / x);
        Console.WriteLine("Selector query created");
        try { _ = risky.ToArray(); }
        catch (DivideByZeroException) { Console.WriteLine("Selector: at enumeration"); }
    }
}

public sealed class Box
{
    public Box(int value) { Value = value; }
    public int Value { get; set; }
}

Verified output

After declaration: 0
First: 2, 4
Calls: 4
Second: 2, 4, 6
Calls: 9
Counts: snapshot 5, source 6
Shared element: 20
Null source: immediate
Selector query created
Selector: at enumeration

Read the counts carefully

Declaration makes zero predicate calls. The first traversal reads four elements. Adding 6 before the second traversal makes it read five more, so the cumulative count is nine. The source is changed between traversals, never during an active traversal. These instrumentation side effects reveal execution; ordinary business predicates should preferably avoid side effects.

ToArray captures the sequence's membership into an array at that time; it does not recursively clone referenced objects. Adding 8 later changes only the list's membership, while editing the existing Box is visible through both owners. The new array itself is mutable. If stable element values are required, project into appropriate immutable values or deliberately copy them. See ToArray.

Where should materialization happen?

Materialize a finite result when multiple consumers must reuse the same computed membership, the original source will go away, or recomputation is costly or undesirable. Keep the query deferred when one streaming pass is enough and producing everything would waste memory. Avoid inserting ToList after every stage: that adds eager passes and intermediate collections without automatically improving clarity or speed.

With grouping, distinguish reusing a captured group from enumerating the original GroupBy query again. Repeating the original query can rebuild its lookup. Whether to retain grouped results depends on source cost, reuse, memory, freshness needs, and measurement; a method name alone cannot decide.

Solved practice: replace nested Any with an index

Problem: retain each candidate if any allowed identifier matches it. Profile the repeated scan, build an index, and verify that the result has the same meaning.

Contract: both arrays and their string entries are non-null and remain unchanged during the operation. Matching is ordinal case-insensitive. Candidate order and duplicate candidates must survive; duplicate allowed identifiers must not multiply output rows. This is a membership filter, not a join that produces every matching pair.

using System;
using System.Collections.Generic;
using System.Linq;

public static class Program
{
    public static void Main()
    {
        string[] candidates = { "A", "b", "A", "missing" };
        string[] allowed = { "a", "B", "b" };
        var comparer = StringComparer.OrdinalIgnoreCase;
        var scanned = Scan(candidates, allowed, comparer, out int comparisons);
        var indexed = Indexed(candidates, allowed, comparer, out int builds, out int probes);
        Console.WriteLine($"Scan: {string.Join(", ", scanned)}");
        Console.WriteLine($"Index: {string.Join(", ", indexed)}");
        Console.WriteLine($"Same: {scanned.SequenceEqual(indexed)}");
        Console.WriteLine($"Scan equality calls: {comparisons}");
        Console.WriteLine($"Index add attempts: {builds}");
        Console.WriteLine($"Index membership calls: {probes}");
    }

    public static string[] Scan(string[] candidates, string[] allowed,
        IEqualityComparer<string> comparer, out int comparisons)
    {
        ArgumentNullException.ThrowIfNull(candidates);
        ArgumentNullException.ThrowIfNull(allowed);
        ArgumentNullException.ThrowIfNull(comparer);
        int count = 0;
        var result = candidates.Where(item => allowed.Any(key =>
        {
            count++;
            return comparer.Equals(item, key);
        })).ToArray();
        comparisons = count;
        return result;
    }

    public static string[] Indexed(string[] candidates, string[] allowed,
        IEqualityComparer<string> comparer, out int builds, out int probes)
    {
        ArgumentNullException.ThrowIfNull(candidates);
        ArgumentNullException.ThrowIfNull(allowed);
        ArgumentNullException.ThrowIfNull(comparer);
        var index = new HashSet<string>(comparer);
        foreach (string key in allowed) index.Add(key);
        int count = 0;
        var result = candidates.Where(item =>
        {
            count++;
            return index.Contains(item);
        }).ToArray();
        builds = allowed.Length;
        probes = count;
        return result;
    }
}

Verified output

Scan: A, b, A
Index: A, b, A
Same: True
Scan equality calls: 7
Index add attempts: 3
Index membership calls: 4

The measured work and why the rewrite is equivalent

Any short-circuits on its first match. A costs one equality call, b costs two, the second A costs one, and missing costs three: seven total. The indexed version makes three Add attempts and four Contains calls. Those are different kinds of operations: a hash lookup can perform hashing and multiple equality checks internally, so 3 + 4 is not a claim of seven equal-cost machine operations or a measured speedup.

The set collapses duplicate allowed keys under the same comparer. Filtering still walks the original candidates, preserving their order and repeated A. Enumerating the set as the result instead would lose that meaning. Changing the comparer during the rewrite would also change the answer. See Any and HashSet.

For n candidates and m allowed keys, a no-match repeated scan performs n × m comparisons. Building the set once and filtering is expected O(m + n) with O(u) extra index entries for u distinct allowed keys, assuming bounded identifier length and ordinary hash distribution. The returned array also takes O(r) space for r matches. Collisions, allocation, small inputs, and early scan matches can change which approach is quicker. Reusing an index can help, but it must be refreshed if the allowed-data contract changes.

Optional timing lab: measure this workload, not a universal law

The counts above are exact. This separate program measures elapsed time for a larger, fixed string workload: 2,000 candidates and 1,000 allowed keys, with 1,000 matches. It includes index construction and array materialization, warms both methods, alternates their order, verifies every result, and reports the median of four samples. Run in Release mode without a debugger.

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Globalization;
using System.Linq;

public static class Program
{
    public static void Main()
    {
        string[] candidates = Enumerable.Range(0, 2000).Select(i => $"id{i}").ToArray();
        string[] allowed = Enumerable.Range(1000, 1000).Select(i => $"ID{i}").ToArray();
        string[] expected = candidates.Skip(1000).ToArray();
        var comparer = StringComparer.OrdinalIgnoreCase;
        string[] Scan() => candidates
            .Where(item => allowed.Any(key => comparer.Equals(item, key))).ToArray();
        string[] Index() => candidates
            .Where(new HashSet<string>(allowed, comparer).Contains).ToArray();

        void Verify(string[] result)
        {
            if (!result.SequenceEqual(expected))
                throw new InvalidOperationException("Results differ.");
        }
        double Measure(Func<string[]> run)
        {
            var timer = Stopwatch.StartNew();
            string[] result = run();
            timer.Stop();
            Verify(result); // Verification is outside the timed region.
            return timer.Elapsed.TotalMilliseconds;
        }

        Verify(Scan());
        Verify(Index());
        var scanTimes = new List<double>();
        var indexTimes = new List<double>();
        for (int round = 0; round < 4; round++)
        {
            if (round % 2 == 0)
            {
                scanTimes.Add(Measure(Scan));
                indexTimes.Add(Measure(Index));
            }
            else
            {
                indexTimes.Add(Measure(Index));
                scanTimes.Add(Measure(Scan));
            }
        }
        Console.WriteLine($"Runtime: {Environment.Version}");
        Console.WriteLine($"Matches: {expected.Length}");
        Console.WriteLine("Samples per method: 4");
        Console.WriteLine($"Scan samples ms: {Samples(scanTimes)}");
        Console.WriteLine($"Index samples ms: {Samples(indexTimes)}");
        Console.WriteLine($"Scan median ms: {Format(Median(scanTimes))}");
        Console.WriteLine($"Index-build-and-filter median ms: {Format(Median(indexTimes))}");
    }

    private static double Median(List<double> values)
    {
        values.Sort();
        return (values[1] + values[2]) / 2;
    }
    private static string Samples(IEnumerable<double> values) =>
        string.Join(", ", values.Select(x => x.ToString("R", CultureInfo.InvariantCulture)));
    private static string Format(double value) =>
        value.ToString("F3", CultureInfo.InvariantCulture);
}

Output contract: Runtime prints the installed .NET 10 runtime version, Matches is 1000, and Samples per method is 4. Raw sample durations are printed so you can recompute both medians. The two elapsed-millisecond values vary by machine and run; there is intentionally no fixed timing output and no assertion that the indexed version must win. The clock is Stopwatch.

One observed timing run

Measured in Release mode on Windows x64 with .NET SDK 10.0.401 and runtime 10.0.12. This is one run of the finite 2,000-candidate, 1,000-key workload above. The raw samples and recomputed medians are observed measurements; timing values vary and do not define a pass condition or a general speed guarantee.

Runtime: 10.0.12
Matches: 1000
Samples per method: 4
Scan samples ms: 10.0686, 10.3753, 10.6502, 11.4244
Index samples ms: 0.4372, 0.1227, 0.1569, 0.1271
Scan median ms: 10.513
Index-build-and-filter median ms: 0.142

This small lab is useful for checking whether the rewrite helps this case. It is not a statistically rigorous benchmark suite. Runtime optimizations, warm-up, garbage collection, scheduling, input sizes, match positions, and comparer costs affect results. Record the environment and repeat with realistic data and distributions before making a performance promise. Do not time only query construction: it can omit nearly all the work.

Database boundary and failure modes

An IQueryable provider interprets expressions for its data source. Enumerating a remote query again can issue another query. Materializing too early can transfer unnecessary rows and move later work into memory. Inspect provider behavior and generated queries rather than applying this in-memory benchmark as a database rule. See IQueryable.

• Repeated expensive enumeration: name the reuse boundary and decide whether results should be fresh or captured.
• Unintentional nested scans: inspect membership checks inside predicates; preserve equality, duplicates, ordering, and freshness when introducing an index.
• Hidden full-input work: a deferred operator can still buffer at enumeration. Take after grouping does not undo the grouping cost.
• Accidental semantic changes: removing duplicate candidates, using a different comparer, or moving Take before ranking answers a different question.

Interview check with answers

What does deferred execution buy you? It composes work and can avoid processing elements nobody consumes. It also postpones callback work, can repeat it, and may observe a later source state. It does not mean every error is delayed.

When should you materialize grouped results? When the required reuse and stability justify retaining them. Consider source cost, memory, and freshness, then measure. GroupBy already buffers during enumeration; it is not automatically cached across separate enumerations of the query.

How do you prove the index rewrite is correct? Compare results with duplicates, differing case, missing keys, and empty inputs. Assert exact output order, not only set equality. Then measure costs separately from correctness.

Sources

• Where

• Select

• GroupBy

• ordering

• LINQ-to-Objects execution classification

• ToArray

• Any

• HashSet

• Stopwatch

• IQueryable