1 of 97

Beyond OAuth2: End to End Microservice Security

William Tran, Principal Software Engineer, Pivotal

Twitter: @fivetenwill

github.com/william-tran/microxchg2017

2 of 97

About Me

  • Spring user since 2.0
  • One of the first customers of AWS (EC2, S3)
  • Currently with Pivotal
  • Contributor to Cloud Foundry UAA and Spring Cloud
  • Currently working on Spring Cloud Services for Pivotal Cloud Foundry
  • Based in Toronto, Canada

3 of 97

The What and Why of Microservices

4 of 97

What do microservices look like?

  • Mostly HTTP based
    • Sometimes invoked via messaging (eg RabbitMQ, Kafka)
  • Independently deployable
  • Single responsibility
  • Separate processes

5 of 97

What do microservice organizations look like?

  • More teams
    • Smaller in size
    • More independent
  • More trust boundaries?
    • Depends on the organization
    • e.g. team A cannot deploy team Bs services

6 of 97

Why microservices? Speed to production.

7 of 97

The Challenges of Securing Microservices

8 of 97

Mo' Processes, Mo' Problems.

9 of 97

OAuth2 to the rescue?

App

Auth Server

Resource

Browser makes a request to the App

/

10 of 97

OAuth2 to the rescue?

App redirects to Auth Server. User authenticates.

/authorize

App

Auth Server

Resource

11 of 97

OAuth2 to the rescue?

Auth Server redirects back to App with an Authorization Code.

/callback?code=xyz

App

Auth Server

Resource

12 of 97

OAuth2 to the rescue?

App gives Authorization Code to Auth Server

/oauth/token?code=

App

Auth Server

Resource

13 of 97

OAuth2 to the rescue?

Auth Server responds with JWT representing the user. App validates JWT.

App stores JWT in session. Browser's session is now authenticated.

JWT

App

Auth Server

Resource

14 of 97

OAuth2 to the rescue?

App uses JWT to request the Resource on behalf of User.

Resource validates JWT.

/api (JWT in header)

App

Auth Server

Resource

15 of 97

OAuth2 to the rescue?

data

homepage

Resource responds with data, and App renders page.

App

Auth Server

Resource

16 of 97

The limits of OAuth2

/feature2

Browser makes another request for some feature.

App

Auth Server

Resource

17 of 97

The limits of OAuth2

/api/2 (JWT in header)

App uses JWT stored in session to make another request to Resource.

App

Auth Server

Resource

18 of 97

The limits of OAuth2

A

... (same JWT in headers)

Resource validates JWT.

Resource reuses JWT for downstream requests.

B

App

Auth Server

Resource

19 of 97

The limits of OAuth2

App

Auth Server

Resource

A

... (same JWT in headers)

And the same JWT continues to be propagated and reused.

B

C

D

20 of 97

One token to rule them all!

21 of 97

Token Propagation

  • The token is too powerful
  • Can be used to do anything to the system as that user
    • Until it expires
  • Token leakage is a big deal
  • Internal fraud is easy

22 of 97

New Tokens via Client Credentials Grant

App

Auth Server

Resource

/api/2 (JWT in header)

App uses JWT stored in session to make another request to Resource.

23 of 97

New Tokens via Client Credentials Grant

App

Auth Server

Resource

/oauth/token

Resource requests a new token to use downstream via Client Credentials grant type.

24 of 97

New Tokens via Client Credentials Grant

App

Auth Server

Resource

JWT for Resource

Resource receives a JWT representing itself.

25 of 97

New Tokens via Client Credentials Grant

App

Auth Server

Resource

Resource uses its JWT for downstream requests.

A and B can authenticate the request as coming from Resource.

Resource says it's acting on behalf of the User.

A

... (Resource JWT in headers)

B

26 of 97

Confused deputy attack

  • A program fooled into misusing its authority
    • e.g. when one app fully trusts another by virtue of the app's identity
  • A and B fully trust Resource
    • Where's the proof that Resource is acting faithfully?
    • How can A and B know if the User is actually authorized?
  • If Resource is compromised, A and B will also be compromised

27 of 97

Total (and misplaced) trust in the real world

  • If apps fully trust one another, do teams as well?
    • transitively, perhaps
    • Do orgs really work that way?
  • What happens when you have no trust boundaries?
    • e.g. You don't check the other person's work
    • e.g. You allow a single person to perform multiple critical tasks
    • e.g. You have no Separation of Duties

28 of 97

29 of 97

30 of 97

60%

Of all attackers are "insiders"

  • IBM Security, 2016

31 of 97

Don't let one bad apple spoil the whole bunch.

32 of 97

Confused deputy mitigations

  • Authorize based on more than just the caller's identity
    • The user and their scopes
  • Send both client and user's token?
    • Still vulnerable, the combination is not integrity protected
  • Authorize based on a composite token
    • Rather than multiple tokens that can be re-arranged
  • Authorize based on a call stack
    • No information loss
    • Specify allowable behaviours with high precision

33 of 97

New Tokens via Token Exchange

  • Proposal for a new OAuth2 Grant type
    • https://tools.ietf.org/html/draft-ietf-oauth-token-exchange-07

34 of 97

New Tokens via Token Exchange

App

Auth Server

Resource

/api/2 (JWT in header)

App uses JWT stored in session to make another request to Resource.

35 of 97

New Tokens via Token Exchange

App

Auth Server

Resource

/oauth/token

Resource requests a new token to use downstream via Token Exchange grant type.

Resource provides its own client credentials, the user's JWT it received, and the Service(s) it wants to hit.

36 of 97

New Tokens via Client Credentials Grant

App

Auth Server

Resource

JWT for User + Resource meant for (A,B)

Resource receives a JWT representing both itself and the User, which can be used for A and B

37 of 97

New Tokens via Client Credentials Grant

App

Auth Server

Resource

Resource uses the JWT in requests to A and B

A and B can authenticate the request as coming from Resource, acting on behalf of the User, and meant for those services.

A

... (Resource + User JWT in headers)

B

38 of 97

New Tokens via Token Exchange

  • Given Actor + Subject + Audience, get a new token
    • Policy decision given caller, user, and intent
    • New token expresses caller, user, and intent
  • Given Actor + previous token + Audience, get a new token
    • Policy decision based on delegation chain (aka call stack)
    • Policy decision based on aggregated trust

39 of 97

More trust boundaries,

different problems.

40 of 97

New Tokens via Token Exchange

Pros:

  • User, Client, and call stack
  • Narrow audience and scope
  • Trust boundaries are unambiguous
    • i.e. assume nothing because everything is in front of you
  • Centralized policy management
  • Can be productized

Cons:

  • Network and AS overhead
    • AS bottleneck
  • Security vs Performance
    • Token caching & reuse
      • Longer token expiry
      • Less restricted token
  • Policy Management vs Agility
    • Who knows best how services should interact?

41 of 97

More processes,

more than just auth problems.

42 of 97

Message Security

  • Protect the integrity and confidentiality of data in transit
  • HTTPS/TLS everywhere

43 of 97

HTTPS Everywhere

Shop

Cart

Payments

Shop provides UI and API facade to Cart.

User enters credit card info and hits the confirm payment button.

HTTPS provides confidentiality and integrity between browser and Shop.

/checkout/confirm

8152 4444 1234 5678

44 of 97

HTTPS Everywhere

Shop

Cart

Payments

Cart provides checkout functionality.

Shop passes the CC info to Cart.

HTTPS provides confidentiality and integrity between Shop and Cart.

/cart/confirm

8152 4444 1234 5678

45 of 97

HTTPS Everywhere

Shop

Cart

Payments

Cart calls Payments to settle the order, providing the CC info.

HTTPS provides confidentiality and integrity between Cart and Payments.

/pay-for-order/12345

8152 4444 1234 5678

46 of 97

HTTPS everywhere doesn't solve everything

  • Does Cart really need to see the CC info?
    • Payments is the only service that really uses it
  • Only Point-to-Point confidentiality and integrity are assured
  • Rearrange services so Payments can be directly called by Shop
    • Limits architectural choices
    • Overcomplicates Shop

47 of 97

End-to-End Message Security

  • Cannot be solved at the network layer
  • Necessarily an application layer concern
  • WS-Security "solved" this for XML a decade ago*
    • way too complicated, still had flaws

48 of 97

Solution Goals

  • End to End Message Security
  • Stop assuming trust based solely on caller's identity
  • Limit the effect of token leakage and misuse
  • Break out of the performance vs security tradeoff
    • Eliminate the AS bottleneck

49 of 97

A Possible Solution

50 of 97

Disclaimer

  • Solution is experimental
  • Feedback appreciated!
  • github.com/pivotal-cf-experimental/spring-cloud-services-security
    • Apache 2 Licenced
  • Runs on Pivotal Cloud Foundry (coming soon)
    • Spring Cloud Services Config Server and Service Registry (Eureka)
    • Pivotal Single Sign On
  • Runs locally using OSS Spring Cloud + UAA (coming soon)

51 of 97

JOSE:

  • not just a Spanish name

52 of 97

JOSE to the rescue

  • JSON Object Signing and Encryption
  • Not just for JSON payloads
  • Proposed Standards (May 2015)
    • JSON Web Signature (JWS) RFC-7515
    • JSON Web Encryption (JWE) RFC-7516
    • JSON Web Token (JWT) RFC-7519
  • Thanks Nimbus JOSE + JWT!
    • https://connect2id.com/products/nimbus-jose-jwt/

53 of 97

JOSE for Message Security

  • Use public/private keypairs
    • e.g. RSA, more info: connect2id.com/products/nimbus-jose-jwt/algorithm-selection-guide
  • Sender signs message with private key
    • Integrity and non-repudiation
  • Sender encrypts signed messages with public key(s) of recipient(s)
    • Confidentiality
    • Multiple recipients via JWE JSON Serialization on Nimbus roadmap
  • Recipient decrypts with its private key, verifies with sender's public key

54 of 97

Keypairs from where?

  • Public Key Infrastructure (PKI)
    • Let's Encrypt
    • Netflix Lemur
    • Your own
  • Something simpler?

55 of 97

Service Discovery

  • Decouples service identity from physical networking
    • e.g. //my-service/api instead of http://my-service-123.example.com/api
  • Enables changes to deployment at runtime
  • Shields consumers from changes in deployment
    • e.g. Rolling out new versions

56 of 97

Service Registry

  • e.g. Netflix Eureka, Apache Zookeeper, Hashicorp Consul
    • We'll use Eureka
  • Services register with the registry on startup
    • service name, how to reach it (hostname, IP + port)
    • Other metadata for smart routing decisions e.g. version

57 of 97

Service Registry and Trust

  • Services use the registry to look each other up
    • Services must have a high level of trust with the registry to trust the routing information it gets from it
  • Registry receives registration requests from services
    • Registry must have high level of trust with services to receive faithful registration requests from them

58 of 97

Trusted Service Registries and Service Identity

  • Trust can be established via authentication and access control
    • Each service gets its own credentials for registration
    • Prevent services from modifying another service's records
    • Already done in Spring Cloud Services
  • Register public key as well as routing info
    • Generate keypair at startup
    • Private key can truly be private
  • Service Registry can now be used to prove Service Identity

59 of 97

Service Discovery = Identity + Location.

It must be trustworthy.

60 of 97

Sign All Of The Things

  • The sender signs with its private key
  • HTTP
    • Request
    • Response
    • Headers (the ones that matter)
  • Messaging
    • Headers
    • Message bodies

61 of 97

Verify All Of The Things

  • Receiver uses the registry to map sender → public key
  • Use the public key to verify payloads
  • Payloads are traceable to the individual service that registered

62 of 97

Encrypt Some Of The Things

  • Sender uses the registry to map receiver → public key
  • Use public key to encrypt payloads
    • sign, then encrypt
    • multiple receivers possible via JWE JSON Serialization
    • encrypt entire bodies, objects or single fields
  • Use it when needed

63 of 97

Message Security: that was easy

64 of 97

Are we there yet?

  • End-to-end message confidentiality with JWE
  • Message integrity via JWS
    • Service authentication "for free"
  • Still need to deal with authorization:
    • Assert and authorize call stack
      • Stop assuming trust based solely on caller's identity
    • Limit the effect of token leakage and misuse
    • Break out of the performance vs security tradeoff

65 of 97

Decentralize & Embed

All Of The Things

66 of 97

Self Issued JWT

  • Services authenticate with JWTs they create and sign themselves
    • JWT is just a specific use case of JWS
    • Services authenticate by verifying with sender's public key
  • Create as many as you need w/o network overhead

67 of 97

Single Use JWT

  • Short expiry
  • JTI can prevent replay attacks
  • Unambiguous intent
    • Express the intended recipient and operation to be performed
  • Very limited power
    • Can only be used for intended purpose and only once
  • Services only accept these JWTs
    • Reject the bare user token → user token has no power on its own

68 of 97

Nested, Self Issued JWT

  • Incoming JWTs can be nested in another JWT for use downstream
    • Call stack expressed as nested JWTs
  • Verifiable chain of custody
    • Like an audit trail
    • Call stack verified by verifying each JWT recursively
  • Unbreakable chain of custody
    • Sender and receiver is encoded in the chain
    • Chains are wrapped in another JWS, which doesn't get propagated
    • Chain truncation can be detected

69 of 97

Chain Truncation

Cart JWT

Shop JWT

Shop

Cart

Payments

User JWT

Shop JWT

User JWT

1

2

Shop JWT

User JWT

3

70 of 97

Preventing Chain Truncation With JWS Envelope

Payments Envelope

Cart Envelope

Shop Envelope

Cart JWT

Shop JWT

Shop

Cart

Payments

User JWT

Shop JWT

User JWT

1

2

Shop JWT

User JWT

3

71 of 97

Example Nested JWT + Service Registry

{

"alg": "RS256",

"typ": "JWT"

}

{� "jti" : "ddd",� "sub" : "2938409238502935",� "scope" : [ "openid", "shop.customer" ],� "cid" : "shop",� "grant_type" : "authorization_code",� "user_name" : "marissa",� "iat" : 1477523159,� "exp" : 1477566359,� "iss" : "https://uaa.shop.com/oauth/token",� "aud" : [ "openid", "shop" ]�}

[signature of JWT]

Token received by Shop from AS

{

"alg": "RS256",

"typ": "JWT"

}

{

"ini": "eyJhbGciOi...",

"jti": "eee",

"iss": "123",

"aud": "CART",

"op": "POST /cart/confirm",

"iat": 1477523259,

"exp": 1477523264,

"bdy": true

}

[signature of JWT]

Token received by Cart

{

"alg": "RS256",

"typ": "JWT"

}

{

"jwt": "eyJhbGciOi...",

"jti": "fff",

"iss": "456",

"aud": "PAYMENTS",

"op": "POST /pay-for-order/12345",

"iat": 1477523260,

"exp": 1477523274,

"bdy": true

}

[signature of JWT]

Token received by Payments

Instance ID

App Name

Location

Metadata

123

SHOP

10.0.0.4:12380

public_key: …

456

CART

10.0.0.5:45034

public_key: ...

789

PAYMENTS

10.0.0.6:25303

public_key: …

72 of 97

Externalized Policy, Embedded Decisionmaking

  • Services make authorization decisions on their own
  • Policy as externalized configuration
    • Not hard-coded into services
    • e.g. Spring Cloud Config Server
  • No additional calls required for authorization
    • Performance FTW
  • Policies can be flexible
    • Allow services to evolve

73 of 97

Example Policy (for Payments)

security:

policy:

for-each-of-the-following:

- enforce-the-first-matching-rule:

- tokens-that:

have-operation:

matching: "POST /pay-for-order/.*"

must:

have-initial-token:

where-scope-contains:

value: shop.customer

come-from:

- app-name: CART

- app-name: SHOP

via-operation:

equal-to: POST /cart/confirm

74 of 97

Example Nested JWT + Service Registry

{

"alg": "RS256",

"typ": "JWT"

}

{� "jti" : "ddd",� "sub" : "2938409238502935",� "scope" : [ "openid", "shop.customer" ],� "cid" : "shop",� "grant_type" : "authorization_code",� "user_name" : "marissa",� "iat" : 1477523159,� "exp" : 1477566359,� "iss" : "https://uaa.shop.com/oauth/token",� "aud" : [ "openid", "shop" ]�}

[signature of JWT]

Token received by Shop from AS

{

"alg": "RS256",

"typ": "JWT"

}

{

"ini": "eyJhbGciOi...",

"jti": "eee",

"iss": "123",

"aud": "CART",

"op": "POST /cart/confirm",

"iat": 1477523259,

"exp": 1477523264,

"bdy": true

}

[signature of JWT]

Token received by Cart

{

"alg": "RS256",

"typ": "JWT"

}

{

"jwt": "eyJhbGciOi...",

"jti": "fff",

"iss": "456",

"aud": "PAYMENTS",

"op": "POST /pay-for-order/12345",

"iat": 1477523260,

"exp": 1477523274,

"bdy": true

}

[signature of JWT]

Token received by Payments

Instance ID

App Name

Location

Metadata

123

SHOP

10.0.0.4:12380

public_key: …

456

CART

10.0.0.5:45034

public_key: ...

789

PAYMENTS

10.0.0.6:25303

public_key: …

75 of 97

Advances use cases

  • Custom claims in Self Issued JWT
    • lets services make their own claims
    • claims propagate and can be verified by any downstream app at any depth
  • Custom assertions
    • For fine grained authorization
    • Get signed object from one services's response
    • Give it to another service to use as an assertion

76 of 97

Demo Time!

77 of 97

Demo goals

  • Demonstrate core verifications
    • Chain of custody
    • Token expiry
    • Token intent (aud, op)
  • Demonstrate flexible invocation path policies

78 of 97

Coming Soon...

  • Integration with Gateway and @EnableOAuth2Sso
  • Integration with Eureka, RestTemplate
  • Integration with Spring Security
  • Integration with Spring MVC

79 of 97

Coming Maybe...

  • Encryption?
    • Java Library for JWE JSON Serialization required for general support
    • .Net needs it too
    • Node.js does it! https://www.npmjs.com/package/node-jose
  • No HTTP, no problem
    • pub-sub via Spring Cloud Stream coming soon

80 of 97

Watch this space:

github.com

/pivotal-cf-experimental

/spring-cloud-services-security

81 of 97

Stay In Touch

William Tran

Principal Software Engineer

Pivotal

github.com/william-tran

linkedin.com/in/fivetenwill

@fivetenwill

wtran@pivotal.io

82 of 97

The Gory Details

83 of 97

Risks

  • Token Size
    • 10 iterations = 55 KB
  • Collusion
    • A says they're doing POST /foo in the token
    • A actually POSTs /bar to B
    • B doesn't validate operation in token with actual operation
    • B sends wraps A's token with its own, and sends to C
    • C believes A POSTed /foo to B

84 of 97

The Authorization Server

  • e.g. Gateway app gets token for user via authcode grant type
  • AS must issue a "composite JWT" describing the user and client
    • sub: user's GUID
    • scope: what the token can be used for
    • cid: client ID of gateway app
  • e.g. Cloud Foundry UAA

85 of 97

A new token for every request

  • Gateway creates its own JWT, including the user's token
    • ini: the initial token from the AS
  • Gateway signs the resulting nested JWT with its private key
  • Gateway sends JWT with downstream request

86 of 97

Preventing reuse and replay with jti and exp

  • Take advantage of single use token semantics
  • jti: JWT ID, a random string aka nonce
  • exp: expiry time
    • make it short, but account for clock skew
  • Validate token
    • quick check: exp is not in the past
    • rigorous check: jti in datastore

87 of 97

Guaranteed non replay

  • Use Redis with ttl = exp
    • Network hop
  • Use a bloom filter locally first
  • Build it into your system of record
    • eg save the jti with the "transaction" in a unique column
  • Do this when it really matters
    • Performance vs security

88 of 97

Limit the token's power

  • Audience (aud) claim
    • The token's intended recipient
    • Or recipients in pub/sub messaging
  • Operation (op) claim
    • The intended operation to perform on the recipient(s)
    • e.g. HTTP Method and Path
    • e.g. Spring Cloud Stream destination
  • Intent of the sender is clear and unambiguous to the receiver
  • Token can only be used for its intended purpose

89 of 97

Link the Token to the JWS Body

  • In the JWT, bdy: true if there should be a body
  • In the JWS body's header, jti: [JTI of token]
  • Token cannot be separated from body without detection

90 of 97

Custom Claims

  • Because you can
  • Any claim added to a token will be visible to any downstream service
    • Regardless of depth
    • They will know who added the claim

91 of 97

Example Nested JWT

{

"alg": "RS256",

"typ": "JWT"

}

{

"ini": "eyJhbGciOi...",

"jti": "0120124b-5f4c-4791-99c4-28bc062eb9ef",

"iss": "6366d4ae-7f65-4b73-83f5-8ec0c84e8200",

"aud": "FUNDS-TRANSFER",

"req": "POST /api/funds-transfer",

"iat": 1477523159,

"exp": 1477523164,

"bdy": true,

"ip": "123.123.123.123"

}

eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJqd3QiOiJleUpoYkdjaU9pLi4uIiwianRpIjoiYWVkOTZkODYtODU2MC00MmRmLTg4NjAtNzNkMzE4ZWNhOWIwIiwiaXNzIjoiOTc0MWRmMWItYTY1Mi00ZTY2LTg0MzEtMDAyNGZkYzFlNDkyIiwiYXVkIjoiRlVORFMtVFJBTlNGRVIiLCJyZXEiOiJHRVQgL2FwaS9hY2NvdW50cz90eXBlPXRyYWRpbmciLCJpYXQiOjE0Nzc1MjMxNTksImV4cCI6MTQ3NzU2NjM1OSwiaXAiOiIxMjMuMzIxLjEyMy4zMjEifQ.0AwnQzABbqQM2qpawQW9fwL3fzTk1ZfHHSHRMeuY1ogx-ic-2_HpU5PqGDTo-jq9q1Ep7C6FZQ4McS7_zr5RGt2YP2P5qc43yD3HM4WQrZ88Ilv-e-ZQzXCSBSzINIXp1JUl6TNQBneMIF5O_BxMLQK-IuqblL2dA_Q9jznr9YxS8qrQ74kCJPlldxjXD-TZEDCuc22hlOgAkJGlAPt7XchGqo4w1D3IiANFxIOVOj3y3kpyds-G8m_yZMIdC6Aqov7FCf4HWXitIZqOdHaJA8R1fnoC54LtO3bOcvwp6Tt-bo1PxjBwkjd9hE1ea5QQ9_hy-PGzSuA1H3sFt06OCQ

92 of 97

Example Corresponding JWS Body

{

"alg": "RS256",

"typ": "JOSE",

"jti": "0120124b-5f4c-4791-99c4-28bc062eb9ef",

"cty": "application/json"

}

{

"account_from": "123",

"account_to": "456",

"amount": "1000"

}

eyJhbGciOiJSUzI1NiIsInR5cCI6IkpPU0UiLCJqdGkiOiI1NjJmNmE3Yi02MmI1LTQwYjAtOTQ3Yi02M2MxOTk1NzdkNjYiLCJjdHkiOiJhcHBsaWNhdGlvbi9qc29uIn0.eyJhY2NvdW50X2Zyb20iOiIxMjMiLCJhY2NvdW50X3RvIjoiNDU2IiwiYW1vdW50IjoiMTAwMCJ9.S1tZw-yGSXRP9zH72RKZ7Vsdl71sGHspWqquxjKbrqp-s7zx2BcjCbvuxKITpvMabB_xQ5EeMrgpKgn4b8YLahQpTshw_m9xxoCVUmo4IMQObUVGBXQuBxQPwDiYl-13CuNgeCpIEm7xCJNEorVhZhmV5FYuS-Qe3SONSgy0SwQLZDta0PAtC2vsaIezUiciErgAmRFa9wjOKS0IGqSeWueuGoBbGx8KoWMio-kacqnTV9b_ROT7tAtMdc8rNRfCAdn72QDtE18nWE7cSDQxu3BKpj8Vod1De1I3N0adJ_7UW5s_KEppDm2y2MelCo0sy2IFYl7MCuB8XDgW4I0_PQ

93 of 97

Fire Away

  • JWT over HTTP in Authorization header
    • bearer eyJhbGciOi…
  • JWS body in body
    • Content-type: application/jose

94 of 97

Repeat

  • Subsequent downstream services do almost the same thing
    • nest the incoming token in "jwt" claim instead of "ini"
    • jwt: a self issued JWT
    • ini: a token from the AS, validated differently
  • Token is built up like layers of an onion

95 of 97

Token Size

  • RSA 2048 signature, initial token from UAA

96 of 97

Token Size

  • Use ECDSA for smaller signatures of equal strength
  • Configure your infrastructure accordingly
  • PWS header limit is 10 MB
  • Spring Boot: server.max-http-header-size

97 of 97