Rules are run at the data source level-of-detail (a row-level calculation). One can think of it as a conditional calculation of a new column done per data source. The results are then aggregated using SUM to create the query result.
Formula are calculated at the visualisation level-of-detail (an aggregate calculation) after summarizing. These are also used to calculate totals and can thus also correctly calculate rate and ratios between other metrics.
To illustrate this, imagine a simple Funnel account with one Facebook and one Google Analytics account. In this account we have created a custom rule based metric for Cost that uses Spend metric for Facebook and for Google Analytics we multiply any affiliate conversion with 1 dollar. There is also a formula based metric, CPC, that divides the custom Cost with Clicks.
To provide a result showing Cost and CPC per Campaign Funnel would:
For Facebook data, apply Cost rules
For Google Analytics data, apply Cost rules
Summarize data from all data sources per Campaign
On the aggregated data apply CPC formula
Re-apply CPC formula for totals or other visualizations
Note that formula based metrics in visualizations can not always be calculated in correct order after other formulas are calculated, depending on where this formula is applied. It is therefore not possible to use a formula in another formula. However, since rule based metrics are calculated before formulas they can depend on other aggregatable metrics and be used in formulas.