HTTP signature claimed to be invalid

Natureshadow · April 13, 2023, 12:25pm

Hi,

currently implementing ActivityPub, and testing it against Mastodon, I am banging my head at the HTTP signature not verifying correctly.

Here is what the Mastodon instance replies:

{
 'error': 'Verification failed for clitest2@vocatadev.pagekite.me https://vocatadev.pagekite.me/users/clitest2 using rsa-sha256 (RSASSA-PKCS1-v1_5 with SHA-256)',
 'signed_string': '(request-target): post /users/pinguin/inbox\nhost: floss.social\ndate: Thu, 13 Apr 2023 12:07:38 GMT\ndigest: SHA-256=3lpN7mh9QKgOz4SJKXrD+hMNS9E7yex+QiJ9aRZ2LB8=',
 'signature': 'E1y8Tg7ew5pWFsDJdXOrC5UfbgC0gVct6RJnLx9V44wklimkm3Mry9trNHBJNtvHJnlZID0URpYuDI7NOWb0d6xxP51jmB9oIwzYnlM+IOZNvzfHCEK05NasoVSakeOGbLWUURpjGYzWVPFB+4Ys7/Yw5iTShsgsaP+nvvWasNAkPPQD34IK2Tfin09Pjh5DQGnsITMLjKbzhERVGzGrAFT2msmm6lCUT6JpLVDxmYLB5ewehmWgqKSEt7x9N+8eMcjAdhbMKTEpovrmQDdMs0Gd9edJBrcvopXp6zLvRTFu9/NB1A92DZOIFt2FD8nk2/g46ycJ3Wd/inuz/meDrQ=='
}

And here is my actor as represented when retrieving it over HTTP:

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "id": "https://vocatadev.pagekite.me/users/clitest2",
  "type": "Person",
  "inbox": "https://vocatadev.pagekite.me/users/clitest2/inbox",
  "https://w3id.org/security#publicKey": {
    "id": "https://vocatadev.pagekite.me/users/clitest2#aiyPjWeV8bjVQWU5xyZ6qA",
    "https://w3id.org/security#owner": "https://vocatadev.pagekite.me/users/clitest2",
    "https://w3id.org/security#publicKeyPem": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArI46FaB6S35L8UjOuHO3\nAsolH0HNCjeFtbZIYJ2Qf7SrSp6/F4ZHGptNKADe1wt+5Smx7m/clDV5REyR9JXr\naxsEOymeza+ZRkzAXbC4VZNZ+uhk9L2mPlwmTBaF+snjqHW2CsyfSGZ7RdhzPt1I\nk8d4yzoMFxC3p8HTken/zdSQNTRObqFWRSgKf7bOYL6Acpu2mE7aO4AHE/ZVTPoU\nsQ8bm4eBik1MqLDS1Sg+bC7h7ID64sf1w1vNthdqlW8B1kqFocxtO9O/M9jxITcl\n96V0lklvTAn1MHtWlqW2bgTWImQZp55+qhSSleozTQt9jHe89LCFbn4WcCH8jnWz\n9wIDAQAB\n-----END PUBLIC KEY-----\n"
  },
  "followers": "https://vocatadev.pagekite.me/users/clitest2/followers",
  "following": "https://vocatadev.pagekite.me/users/clitest2/following",
  "name": "Vocata CLI Test User",
  "outbox": "https://vocatadev.pagekite.me/users/clitest2/outbox",
  "preferredUsername": "clitest2"
}

I have verified the signature myself both through my own implementation and by stuffing it into the first HTTP signature verification snippet I found on StackOverflow, and both are happy with it ;).

Also, I see Mastodon do all the dereferencing:

INFO:     ::ffff:141.95.205.35:0 - "GET /users/clitest2 HTTP/1.1" 200 OK
INFO:     ::ffff:141.95.205.35:0 - "GET /users/clitest2 HTTP/1.1" 200 OK
INFO:     ::ffff:141.95.205.35:0 - "GET /webfinger?resource=acct:clitest2@vocatadev.pagekite.me HTTP/1.1" 200 OK
INFO:     ::ffff:141.95.205.35:0 - "GET /users/clitest2/outbox HTTP/1.1" 200 OK
INFO:     ::ffff:141.95.205.35:0 - "GET /users/clitest2/following HTTP/1.1" 200 OK
INFO:     ::ffff:141.95.205.35:0 - "GET /users/clitest2/followers HTTP/1.1" 200 OK
INFO:     ::ffff:141.95.205.35:0 - "GET /users/clitest2 HTTP/1.1" 200 OK

And if I read mastodon/signature_verification.rb at af49d93fd6168c089530240a9ab4eccb975b8c42 · mastodon/mastodon · GitHub correctly, the error message means that everything (retrieving actor and key, etc.) was already successful, and what is failing ist the verification of the signature itself.

Any hints on how to debug that, or even what could be the issue?

I might set up my own Mastodon instance and get it to print debug logs…

Thanks,
Nik

helge · April 13, 2023, 4:30pm

Can you check that the message is the same. It should be:

(request-target): post /path/to/resource
host: floss.social
date: Thu, 13 Apr 2023 12:07:38 GMT
digest: SHA-256=3lpN7mh9QKgOz4SJKXrD+hMNS9E7yex+QiJ9aRZ2LB8=

without a new line at the end.

Natureshadow · April 13, 2023, 8:58pm

Yep, that’s the case.

Full output on my side:

           DEBUG    Signing header for POST request to                                      federation.py:71
                    https://floss.social/users/pinguin/inbox                                                
           DEBUG    Adding (request-target) pseudo-header to signature                      federation.py:93
           DEBUG    Adding Host header to request                                           federation.py:89
           DEBUG    Adding host header to signature                                         federation.py:99
           DEBUG    Adding Date header to request                                           federation.py:81
           DEBUG    Adding date header to signature                                         federation.py:99
           DEBUG    Adding Digest header to request                                         federation.py:84
           DEBUG    Adding digest header to signature                                       federation.py:99
           DEBUG    Signing header: (request-target): post /users/pinguin/inbox            federation.py:108
                    host: floss.social                                                                      
                    date: Thu, 13 Apr 2023 20:56:25 GMT                                                     
                    digest: SHA-256=AfSNBRrUo6SFjOFUEjWX7YwbCFooYwM79UOgWWT4icM=                            
           DEBUG    Created signature header                                               federation.py:122
                    keyId="https://vocatadev.pagekite.me/users/clitest3#7i5wToe7t5FSa7BZKB                  
                    Yfou",algorithm="rsa-sha256",headers="(request-target) host date                        
                    digest",signature="mceeOjqm65vBIC1dfZyJxLC+uy74FUMl09Y0jy8NsB9c175QEg1                  
                    hI/tQijPiHfhXeVWyzk7gE94H1HEMkdHaECnYvBNI/B+JDa+IDJ7gTsOEbmys/lf/nSTZM                  
                    DojQZ0IzFlqTwHfy8sKM8AuVihokzhus184sFho0BEFGyskJRvFdKmDol8H/BPvIaCfrtn                  
                    41pwxHLh0gQXk+HPJLM4jh9ZrW77iiMRFHTL7yFE8aawQXrnNolP2ZJW0OS3rCg56aLhva                  
                    HmRlG8iqLSQuZEIQlAFFrFC8oUsRBI7NX3sI/9wXu8xEzqYp91GJkpj+041X0MTQ81Dgks                  
                    YGeQ7sYU+uQ=="                                                                          
[22:56:27] DEBUG    https://floss.social:443 "POST /users/pinguin/inbox HTTP/1.1" 401  connectionpool.py:456
                    None                                                                                    
           ERROR    Request failed with error: {'error': 'Verification failed for          federation.py:166
                    clitest3@vocatadev.pagekite.me '                                                        
                              'https://vocatadev.pagekite.me/users/clitest3 using                           
                    rsa-sha256 '                                                                            
                              '(RSASSA-PKCS1-v1_5 with SHA-256)',                                           
                     'signature':                                                                           
                    'mceeOjqm65vBIC1dfZyJxLC+uy74FUMl09Y0jy8NsB9c175QEg1hI/tQijPiHfhXeVWyz                  
                    k7gE94H1HEMkdHaECnYvBNI/B+JDa+IDJ7gTsOEbmys/lf/nSTZMDojQZ0IzFlqTwHfy8s                  
                    KM8AuVihokzhus184sFho0BEFGyskJRvFdKmDol8H/BPvIaCfrtn41pwxHLh0gQXk+HPJL                  
                    M4jh9ZrW77iiMRFHTL7yFE8aawQXrnNolP2ZJW0OS3rCg56aLhvaHmRlG8iqLSQuZEIQlA                  
                    FFrFC8oUsRBI7NX3sI/9wXu8xEzqYp91GJkpj+041X0MTQ81DgksYGeQ7sYU+uQ==',                     
                     'signed_string': '(request-target): post /users/pinguin/inbox\n'                       
                                      'host: floss.social\n'                                                
                                      'date: Thu, 13 Apr 2023 20:56:25 GMT\n'                               
                                      'digest: '                                                            
                                      'SHA-256=AfSNBRrUo6SFjOFUEjWX7YwbCFooYwM79UOgWWT4icM                  
                    ='}                                                                                     
           ERROR    Failed to push https://vocatadev.pagekite.me/testcreate-priv-to-pingu  federation.py:201
                    to https://floss.social/users/pinguin/inbox

I have access to test Mastodon instance now. If anyone could help me spread debug logging over signature_verification.rb (my head explodes every time I look at Ruby code), especially to find out what OpenSSL has to say when verifying the signature, that would be much appreciated.

Or maybe the issue is obvious enough to someone now…

silverpill · April 13, 2023, 11:13pm

Does Mastodon really support property names like https://w3id.org/security#publicKey? Maybe it expects publicKey, and therefore signature verification fails because it can not retrieve the key from your actor object.

trwnh · April 14, 2023, 2:41am

i would assume not – mastodon doesn’t support JSON-LD fully. it only compacts input that has a signature property pointing to a JSON object. i think this is intended to fix a bug with LD Signatures where the old “identity v1” context doesn’t resolve anymore. see Compact all incoming ActivityPub for future-proofing · Issue #12387 · mastodon/mastodon · GitHub for the issue about compacting input

helge · April 14, 2023, 6:17am

The following test fails, which shows the signature is incorrect. Can you check that the message agrees with the one you are signing?

As bovine is federating fine with Mastodon, I feel it is safe to assume that my code is correct.

from bovine.crypto.signature_parser import Signature
from bovine.crypto.helper import verify_signature


def test_XXX():
    header_string = """keyId="https://vocatadev.pagekite.me/users/clitest3#7i5wToe7t5FSa7BZKBYfou",algorithm="rsa-sha256",headers="(request-target) host date digest",signature="mceeOjqm65vBIC1dfZyJxLC+uy74FUMl09Y0jy8NsB9c175QEg1hI/tQijPiHfhXeVWyzk7gE94H1HEMkdHaECnYvBNI/B+JDa+IDJ7gTsOEbmys/lf/nSTZMDojQZ0IzFlqTwHfy8sKM8AuVihokzhus184sFho0BEFGyskJRvFdKmDol8H/BPvIaCfrtn41pwxHLh0gQXk+HPJLM4jh9ZrW77iiMRFHTL7yFE8aawQXrnNolP2ZJW0OS3rCg56aLhvaHmRlG8iqLSQuZEIQlAFFrFC8oUsRBI7NX3sI/9wXu8xEzqYp91GJkpj+041X0MTQ81DgksYGeQ7sYU+uQ=="""

    result = Signature.from_signature_header(header_string)

    message = """(request-target) post /users/pinguin/inbox
host: floss.social                                                                      
date: Thu, 13 Apr 2023 20:56:25 GMT                                                     
digest: SHA-256=AfSNBRrUo6SFjOFUEjWX7YwbCFooYwM79UOgWWT4icM="""

    public_key_pem = """-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArI46FaB6S35L8UjOuHO3
AsolH0HNCjeFtbZIYJ2Qf7SrSp6/F4ZHGptNKADe1wt+5Smx7m/clDV5REyR9JXr
axsEOymeza+ZRkzAXbC4VZNZ+uhk9L2mPlwmTBaF+snjqHW2CsyfSGZ7RdhzPt1I
k8d4yzoMFxC3p8HTken/zdSQNTRObqFWRSgKf7bOYL6Acpu2mE7aO4AHE/ZVTPoU
sQ8bm4eBik1MqLDS1Sg+bC7h7ID64sf1w1vNthdqlW8B1kqFocxtO9O/M9jxITcl
96V0lklvTAn1MHtWlqW2bgTWImQZp55+qhSSleozTQt9jHe89LCFbn4WcCH8jnWz
9wIDAQAB
-----END PUBLIC KEY-----"""

    assert verify_signature(public_key_pem, message, result.signature)

Natureshadow · April 14, 2023, 7:09am

I would hope that it does. After all, it claims to speak ActivityPub.

And if I read the code linked above correctly, it does find the key, because not finding the key would raise another error earlier.

That’s because the signature and the key mismatch (you took the signature from a later post than the actor object; I rotated IDs and keys in between to rule out caching issues).

helge · April 14, 2023, 7:20am

Mastodon probably cannot handle rotating signatures. Can you provide the public key for the post above? Then I can check if the signature is valid with it.

Natureshadow · April 14, 2023, 7:22am

“Rotated” as in “used another actor with another key pair”. I am sure Mastodon supports that ;).

helge · April 14, 2023, 7:26am

from bovine.crypto.signature_parser import Signature
from bovine.crypto.helper import verify_signature


def test_XXX():
    header_string = """keyId="https://vocatadev.pagekite.me/users/clitest3#7i5wToe7t5FSa7BZKBYfou",algorithm="rsa-sha256",headers="(request-target) host date digest",signature="mceeOjqm65vBIC1dfZyJxLC+uy74FUMl09Y0jy8NsB9c175QEg1hI/tQijPiHfhXeVWyzk7gE94H1HEMkdHaECnYvBNI/B+JDa+IDJ7gTsOEbmys/lf/nSTZMDojQZ0IzFlqTwHfy8sKM8AuVihokzhus184sFho0BEFGyskJRvFdKmDol8H/BPvIaCfrtn41pwxHLh0gQXk+HPJLM4jh9ZrW77iiMRFHTL7yFE8aawQXrnNolP2ZJW0OS3rCg56aLhvaHmRlG8iqLSQuZEIQlAFFrFC8oUsRBI7NX3sI/9wXu8xEzqYp91GJkpj+041X0MTQ81DgksYGeQ7sYU+uQ=="""

    result = Signature.from_signature_header(header_string)

    message = """(request-target) post /users/pinguin/inbox
host: floss.social
date: Thu, 13 Apr 2023 20:56:25 GMT
digest: SHA-256=AfSNBRrUo6SFjOFUEjWX7YwbCFooYwM79UOgWWT4icM="""

    public_key_pem = """-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApx/lFJIPB336+lVQXEyJ
hKdgkzcyCHcYtBHuUL0CtY+IxgKxNw/2J9dJhuj1vITDaTXIBriaPNPMes/9KVlv
RKWnKAAzvZxWD31hlgCDvVTLFz/iA37ZMWgw2P3SpzTMgCeOY3s+Zggc4n2X4rAf
PPGumL3s8+7m9fag5jeLcLo32hHP4wRvKglpkCHj2e368ZXpfa1PM7xxy6JGtyvR
FToxzzi3QYa9tiXOteMvA44Z2grCdykN+HPiBowvPhHvX0xcSHQdKR0l02HL4O8n
U68UWRUALSHqKez+jfiYvrIDTvL/i0IurTsHC0T/jMJdOmxj6Zj3Zxy/E19LZhLO
JQIDAQAB
-----END PUBLIC KEY-----"""

    assert verify_signature(public_key_pem, message, result.signature)

still fails. How do you sign the message? I use:

import base64
import hashlib
import logging

from cryptography.exceptions import InvalidSignature
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import ed25519, padding, rsa
from cryptography.hazmat.primitives.serialization import (
    load_pem_private_key,
    load_pem_public_key,
)


def verify_signature(public_key, message, signature):
    public_key_loaded = load_pem_public_key(public_key.encode("utf-8"))

    assert isinstance(public_key_loaded, rsa.RSAPublicKey)

    try:
        public_key_loaded.verify(
            base64.standard_b64decode(signature),
            message.encode("utf-8"),
            padding.PKCS1v15(),
            hashes.SHA256(),
        )
    except InvalidSignature:
        logger.warning("invalid signature")
        return False

    return True

to verify the key.

Natureshadow · April 14, 2023, 7:58am

My code is here: vocata/federation.py at main - vocata - Codeberg.org

helge · April 14, 2023, 8:23am

Your message is:

(request-target): post /users/pinguin/inbox
host: floss.social
date: Thu, 13 Apr 2023 20:56:25 GMT
digest: SHA-256=AfSNBRrUo6SFjOFUEjWX7YwbCFooYwM79UOgWWT4icM=

You have to remove the colon after (request-target).

Natureshadow · April 14, 2023, 8:31am

This is not true, neither according to the spec linked from the Mastodon docs, nor according to the expected string returned by Mastodon.

helge · April 14, 2023, 8:56am

Oh you are right. Sorry. Then, we have it settled: Your signature is correct, and it must be related to either:

Expanding the security terms, can you try the form:

{
  "@context":[ "https://www.w3.org/ns/activitystreams", "https://w3id.org/security/v1" ],
  "id": "https://vocatadev.pagekite.me/users/clitest2",
  "type": "Person",
  "inbox": "https://vocatadev.pagekite.me/users/clitest2/inbox",
  "publicKey": {
    "id": "https://vocatadev.pagekite.me/users/clitest2#aiyPjWeV8bjVQWU5xyZ6qA",
    "owner": "https://vocatadev.pagekite.me/users/clitest2",
    "publicKeyPem": "-----BEGIN PUBLIC KEY-----\n.....\n-----END PUBLIC KEY-----\n"
  },
  "followers": "https://vocatadev.pagekite.me/users/clitest2/followers",
  "following": "https://vocatadev.pagekite.me/users/clitest2/following",
  "name": "Vocata CLI Test User",
  "outbox": "https://vocatadev.pagekite.me/users/clitest2/outbox",
  "preferredUsername": "clitest2"
}

Something different. Is webfinger working correctly, i.e. can you lookup clitest2@vocatadev.pagekite.me from mastodon?

Natureshadow · April 14, 2023, 9:16am

Yes, that really works.

Thanks for helping out…

(Actually, I somehow suspected that, but as all related chapters in Matodon’s documentation claims it supports JSON-LD, I was mislead to believe that… Hooray for claiming to implement standards and then not doing!)

So, now I just need to do that in a nice (not special-cased) way in my JSON-LD serializer.

trwnh · April 14, 2023, 4:54pm

Mastodon’s documentation should not claim JSON-LD support; where does it say that? I can add a clarification to avoid future confusion.

Natureshadow · April 14, 2023, 8:34pm

Spec Compliance → ActivityPub → first sentence

The second thing after this first sentence should be “Mastodon is not compliant with ActivityPub, it uses JSON documents resembling ActivityPub”.

nightpool · April 14, 2023, 8:50pm

You have misunderstood the ActivityStreams spec. See below:

Serialization

This specification describes a JSON-based [RFC7159] serialization syntax for the Activity Vocabulary that conforms to a subset of [JSON-LD] syntax constraints but does not require JSON-LD processing. While other serialization forms are possible, such alternatives are not discussed by this document.

[…]

An Activity Streams Document is a JSON document whose root value is an Activity Streams Object of any type, including a Collection, and whose MIME media type is " application/activity+json".

Emphasis mine. And the following:

The serialized JSON form of an Activity Streams 2.0 document must be consistent with what would be produced by the standard JSON-LD 1.0 Processing Algorithms and API [JSON-LD-API] Compaction Algorithm using, at least, the normative JSON-LD @context definition provided here. Implementations may augment the provided @context with additional @context definitions but must not override or change the normative context. Implementations may also use additional properties and values not defined in the JSON-LD @context with the understanding that any such properties will likely be unsupported and ignored by consuming implementations that use the standard JSON-LD algorithms. See the Extensibility section for more information on handling extensions within Activity Streams 2.0 documents.

Note that support for extensions can vary across implementations and no normative processing model for extensions is defined. Accordingly, implementations that rely too heavily on the use of extensions may experience reduced interoperability with other implementations.

Mastodon expects the publicKey extension to be present in the @context and therefore compacted down like other ActivityStreams properties. The spec warns that extensions are dependent on individual implementation support.

Handling either form of the property would be a reasonable change request to make, but saying that it “Doesn’t support ActivityPub” because it handles an extension slightly differently is incorrect.