What is the purpose of a "Refresh Token"?

Basically, refresh tokens are used to get new access token.

To clearly differentiate these two tokens and avoid getting mixed up, here are their functions given in The OAuth 2.0 Authorization Framework:

Access tokens are issued to third-party clients by an authorization server with the approval of the resource owner. The client uses the access token to access the protected resources hosted by the resource server. Refresh Tokens are credentials used to obtain access tokens. Refresh tokens are issued to the client by the authorization server and are used to obtain a new access token when the current access token becomes invalid or expires, or to obtain additional access tokens with identical or narrower scope.

Now, to answer your question on why you were still being issued a refresh token instead of just securing an access token, the main reason provided by Internet Engineering Task Force in Refresh tokens is:

There is a security reason, the refresh_token is only ever exchanged with authorization server whereas the access_token is exchanged with resource servers. This mitigates the risk of a long-lived access_token leaking in the "an access token good for an hour, with a refresh token good for a year or good-till-revoked" vs "an access token good-till-revoked without a refresh token."

For a more detailed and complete information of OAuth 2.0 Flow, please try going through the following references:

OAuth 2.0 Flow: Server-side web apps

The OAuth 2.0 Authorization Framework issued by Internet Engineering Task Force (IETF)

SO post - Why Does OAuth v2 Have Both Access and Refresh Tokens?

The refresh token serves at least two purposes. First, the refresh token is a kind of 'proof' that an OAuth2 Client has already received permission from the user to access their data, and so can request a new access token again without requiring the user to go through the whole OAuth2 flow. And second, it helps increase the whole flow of security when compared with a long lived access token. I'll touch on both of these points in a little more detail.

Refresh Tokens as a Means to Not Annoy the User

Let's talk about the first purpose with an example. Suppose you, a User, were using a third party Client web application that wanted to interact with your YouTube account data. Once you grant permission to the Client application to use your YouTube data, would you want the Client app to prompt you for your permission again when its YouTube token expired? What happens if the YouTube token expiry time was something very low, like 5 minutes. It would get a little annoying having the Client application prompt you for your permission at least every 5 minutes! The solution that OAuth2 proposes to this 'problem' is refresh tokens. By using refresh tokens, the access token can remain short-lived (which is desirable in case the access token is leaked or stolen somehow), and the refresh token can remain long(er)-lived, allowing the Client to get a new access token when one expires without requiring the user's permission (again).

But why a refresh token? If the point is to not bug the User with permission requests, then why can't the Client simply say "Hey, Authorization Server, I want another access token. Now!"? Or, "Hey Authorization Server, here is my expired token, give me a new one!". Well, the refresh token serves as a kind of "proof" that the Client at some original point in time was granted access by a User. This "proof" is in the form of the refresh token being digitally signed by the Authorization Server. By the Client presenting a refresh token, the Authorization Server can verify that the Client received, at some point in the past, permission from the User, and the Client does not have to prompt the User again.

Refresh Token as a Means to Increase Security

However, this raises the question, "Well, what happens if the refresh token is leaked or stolen, or simply kept by a malicious Client application that doesn't get rid of it at the user's request? Can't the attacker just continue to use the refresh token to gain a valid access token indefinitely (or until it expires)? This question leads to discussing the second purpose that I mentioned, of refresh tokens contributing to a more secure flow.

The issue that arises with access tokens is that, once acquired, they only ever get presented to the Resource Server (YouTube for example). So if an access token is stolen or compromised, how do you tell the Resource Server not to trust that token? Well, you can't really. The only way to do it would be to change the private signing key on the Authorization Server (the key that signed the token in the first place). I imagine this is inconvenient to do, and in some cases (like Auth0), is not supported.

On the other hand, refresh tokens need to be presented to the Authorization Server frequently, and so if one is compromised, then it is trivial to revoke or deny the refresh token as a whole, and not have to change any signing keys.

@Teyam mention SO post Why Does OAuth v2 Have Both Access and Refresh Tokens? but I prefer the another answer there: https://stackoverflow.com/a/12885823/254109

TL;DR refresh_token does not bring increased security. It's for the purpose to improve scalability and performance. Then, access_token may be stored just in some fast, temporary storage (like memory). It allows the authorization and resource server separation, too.

"So I don't see how this is any more secure than just giving me an Access Token from the start and not bothering with the whole Refresh Token system." I struggled with the same question. The short answer is the refresh token is necessary to assure the credentials have not expired.

An example may help: I have a database that stores your medical records. You consent to sharing your medical records with your spouse. Your spouse uses their Access Token to read your records from my database. Two weeks from now your spouse checks again on your medical records and the refresh token is used to ensure they still have permission (from the authentication server) to view your records. The refresh token bypasses the need for your spouse to re-enter their credentials (username and password) to the authentication server, but it does ensure they still have legitimacy to access the resource. A never expiring Access Token would not know if you had revoked your spouse's rights to access your medical records.

Here is the information from the OAuth 2.0 documentation.

Refresh tokens are used to obtain a new access token when the current access token becomes invalid or expires, or to obtain additional access tokens with identical or narrower scope (access tokens may have a shorter lifetime and fewer permissions than authorized by the resource owner).

  +--------+                                           +---------------+
  |        |--(A)------- Authorization Grant --------->|               |
  |        |                                           |               |
  |        |<-(B)----------- Access Token -------------|               |
  |        |               & Refresh Token             |               |
  |        |                                           |               |
  |        |                            +----------+   |               |
  |        |--(C)---- Access Token ---->|          |   |               |
  |        |                            |          |   |               |
  |        |<-(D)- Protected Resource --| Resource |   | Authorization |
  | Client |                            |  Server  |   |     Server    |
  |        |--(E)---- Access Token ---->|          |   |               |
  |        |                            |          |   |               |
  |        |<-(F)- Invalid Token Error -|          |   |               |
  |        |                            +----------+   |               |
  |        |                                           |               |
  |        |--(G)----------- Refresh Token ----------->|               |
  |        |                                           |               |
  |        |<-(H)----------- Access Token -------------|               |
  +--------+           & Optional Refresh Token        +---------------+

(A) The client requests an access token by authenticating with the authorization server and presenting an authorization grant.

(B) The authorization server authenticates the client and validates the authorization grant, and if valid, issues an access token and a refresh token.

(C) The client makes a protected resource request to the resource server by presenting the access token.

(D) The resource server validates the access token, and if valid, serves the request.

(E) Steps (C) and (D) repeat until the access token expires. If the client knows the access token expired, it skips to step (G); otherwise, it makes another protected resource request.

(F) Since the access token is invalid, the resource server returns an invalid token error.

(G) The client requests a new access token by authenticating with the authorization server and presenting the refresh token. The client authentication requirements are based on the client type and on the authorization server policies.

(H) The authorization server authenticates the client and validates the refresh token, and if valid, issues a new access token (and, optionally, a new refresh token).

access_tokens are more frequently used, and ability to revoke is not very important since they are short lived.

refresh_tokens are less frequently used, and the ability to revoke is crucial since they can be used to generate new access_tokens.

Validating a signed token is less costly, but revoking is difficult.

Validating a token stored in a database is costly, but can be easily revoked.

So, signed keys can be used as access_tokens to improve performance.

Db stored keys can be used as refresh_tokens, to make it easy to revoke them.

If there was no refresh_token, it is hard to find a mechanism which provides low cost validation and easy revocation ability. So refresh_token exists due to performance reasons.

There are at least 3 somewhat related reasons for having short-lived access tokens and long lived refresh tokens.

Bearer Tokens

From the original question:

If I have the Refresh Token, I can ALWAYS use this to get a new Access Token, since it never expires.

Although you may always be able to get a new access token using a refresh token, an attacker may generally not. This is because your use of the refresh token is coupled with some proof of your identity as a client, e.g. by providing your client_secret. Such proof is not needed for access tokens, which are bearer tokens, i.e. simply presenting them is enough.

Having the access token be short-lived mitigates this otherwise unlimited power of access tokens somewhat.

Surface of attack

Access tokens are exchanged with (potentially many) resource servers, which increases the chance of leakage. refresh tokens are only ever exchanged with the authorization server.

Again, the short-livedness of access tokens is at least some level of mitigation.

Revocation

It is possible (and common) to implement access tokens as signed JWTs. In that scenario, any server (that knows the public key of the signing party, which is typically in some well known location) can validate the correctness of the access token independently. This allows for nicely decoupled architectures, in the sense that the resource servers don't have to ask the authorization server about authorization.

The drawback of such a setup is that such tokens cannot be revoked (without anything as drastic as revoking the public key of the authorization server).

By having the access tokens be short lived, they can simply be allowed to run out rather than revoked explicitly.

Using only Access Token is much more risky than using both Access Token and Refresh Token.

For example, if you use only Access Token then set the long expiry date 100 days to Access Token then finally Access Token is stolen by a hacker, the hacker can get the big chance to freely use the Access Token for 100 days as a maximum for bad purposes.

However, if you use both Access Token and Refresh Token then set the shorter expiry date 60 minutes to Access Token and the long expiry date 100 days to Refresh Token then finally Access Token is stolen by a hacker, the hacker can get much smaller chance to freely use the Access Token for 60 minutes as a maximum for bad purposes.

Now, you will think if Refresh Token is stolen. Actually, if Refresh Token is stolen by a hacker, the hacker still can get the big chance to freely use the Refresh Token for 100 days as a maximum for bad purposes. But the probability which Refresh Token is stolen is much less than Access Token is stolen because Refresh Token is used only once every 60 minutes to refresh Access Token(get a new Access Token) while Access Token is used every time you access resources which is much more frequent.

So, you better use both Access Token and Refresh Token.

The access token have a short life span. Once, it expires you need a new access token to access protected resource. One way to obtain a new access token authenticate resource owner again and get authorization grant then obtain access token. However, this will be annoying.

This problem can be solve with refresh token. It has long life span. So you can use it to get new access token without interacting with resource owner.

Well, you may think what is the point of having token with long life-span to obtain another key with short life-span. Well, even refresh token is compromised, attacker cannot just obtain a access token from it. Reason is attacker need the client credentials along with that refresh token.

Therefore, the access token will have a short life-span(reasons for this are available in other answers) to improve the security. To avoid resource owner being annoying whenever the access token expires, OAuth is using refresh token.

It was a good learning experience and getting some idea about the token, refresh token and caching it. However, (I am curious and I am not giving any advice here) can we use the Code which returns after user sign in when using Microsoft Identity platform. CAn we just store the CodeIdToken, and use it to get the new access token whenever there is a need? Because I am thinking we use it to get the access token then should we use to regenerate the access token each time?

 ...
 ResponseType = OpenIdConnectResponseType.CodeIdToken,
...

and

private async Task OnAuthorizationCodeReceived(AuthorizationCodeReceivedNotification context)
    {
        IConfidentialClientApplication clientApp = MsalAppBuilder.BuildConfidentialClientApplication();
        AuthenticationResult result = await clientApp.AcquireTokenByAuthorizationCode(new[] { "User.Read" }, context.Code)
          .ExecuteAsync();
    }
    

refresh_token pattern keeps the OAuth server in control so the server can intervene when something bad happens like access_token and refresh_token getting leaked.

e.g.

if access_token and refresh_token get into the hands of a hacker, access_token will expire shortly, the hacker might try to refresh the token but the server now has the capacity/control to not issue the access_token again (considering the server got information of the leak).