@Karan Shishoo thank you for the link,there are a lot of answers some incompatible with Avalonia, but looks like I figured most of it out:

<NumericUpDown KeyDown="HandleNonNumericInput">

And in the code behind

private void HandleNonNumericInput(object? sender, KeyEventArgs? e)
{
    string? letter = e.KeySymbol;
    bool rejectKey;
    if (string.IsNullOrEmpty(letter))
    {
        rejectKey = true;
    }
    else if (e.Key == Key.Enter || e.Key == Key.Return || e.Key == Key.Escape)
    {
        TextBox? tb = e.Source as TextBox;
        TopLevel? tl = TopLevel.GetTopLevel(this);
        tl!.Focus();
        rejectKey = true;
    }
    else
    {
        rejectKey = !char.IsNumber(letter[0]);
    }
    Debug.WriteLine($"Key: {e.Key}, Symbol: <{letter}>, rejected: {rejectKey}");
    e.Handled = rejectKey;char.IsNumber(e.KeySymbol[0]);
}

I may have forgotten to check for something, if I realize, I'll update. One big problem remains, but that's probably for another question: If at any time the input string is "" there is an exception below the TextBox that doesn't go away.